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Background 


To support states in improving the quality of their Perkins accountability data, in March 2005 the Office of 
Vocational and Adult Education (OVAE), US Department of Education, invited State Directors of Vocational 
Education to submit requests for individualized technical assistance. In response, the New Mexico Public 
Education Department (NMPED), in association with the New Mexico Association for Community Colleges 
(NMACC), submitted an application for conference support to communicate federal and state accountability 
requirements to secondary and postsecondary Perkins sub-recipients. The state also requested help in 
reviewing its current state data collection systems and local performance data, and in establishing a plan of 
action for improving data quality and use. 

Dr. Steven Klein, Associate Director of Policy Analysis and Development, and Ms. Rosio Bugarin, Senior 
Research Associate at MPR Associates, Inc. provided technical assistance services to the state. During 
project planning discussions with Lena Trujillo-Chavez, Bureau Chief for the Career Technical and 
Workforce Education Bureau, MPR researchers identified two activities to meet the state’s technical 
assistance needs. 

^ Present at State Perkins Accountability Meeting 

State officials requested MPR researchers to provide an overview of federal reporting requirements and 
to facilitate breakout sessions at the state’s Perkins Accountability Technical Assistance Workshop, held 
April 5, 2005 in Albuquerque, New Mexico. As an opening activity, MPR researchers summarized 
federal reporting requirements, reviewed state measures and performance targets, and arrayed state 
and local challenges in collecting quality data. 

Following the presentation, MPR researchers split the approximately 175 conference participants into 
secondary and postsecondary work groups and facilitated an hour-long session to identify systemic 
challenges to collecting accurate state and local data. During afternoon breakouts, participants’ critiqued 
current state Perkins measures, proposed changes to improve data precision, and identified pressing 
local technical assistance needs. 


✓ Summarize Meeting Notes 


MPR researchers were asked to synthesize meeting notes and provide NMPED administrators with a 
summary of locally-identified obstacles to collecting quality data and recommendations to improve the 
validity and reliability of reported information. Researchers were also asked to consult with state data 
analysts on system implementation and offer suggestions for improving future state measures and data 
collection tools. 


This report summarizes facilitators’ meeting notes and offers recommendations to support the NMPED in 
structuring technical assistance outreach to secondary and postsecondary career technical education 
(CTE) providers. 
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Synthesis of Secondary Educators’ Comments 


Following a brief presentation on the key components of a quality data collection system, secondary and 
postsecondary educators attended a 1-hour, facilitated breakout session to consider crosscutting issues 
affecting data quality (see Appendix A for a meeting agenda and a copy of session handouts). Participants 
later reconvened for a 1-hour and 45 minute session to identify institutional challenges in collecting Perkins 
data for each core indicator, to recommend strategies for improving statewide reporting, and to identify 
local technical assistance needs. The following section summarizes participants’ observations and offers 
recommendations for improving statewide reporting and technical assistance outreach services. 

Challenge: Measurement definitions and data collection procedures are not clearly defined. 

During both the morning and afternoon breakout sessions, participants reported that they do not always 
understand state definitions and measures or the accepted methodology for collecting and reporting 
outcome data. Participants believe that the absence of clear reporting instructions has undermined the 
quality and consistency of local data. 

Recommendation 

Participants asked that the NMPED review the state’s Perkins definitions, populations, and 
measures, and issue more detailed guidance to assist local educators in collecting and reporting 
information. It was suggested that NMPED staff incorporate revised definitions into its Accountability 
Data System (ADS) manual, and that the state develop technical assistance materials that explain 
which students should be included in each measure and the timing and procedures for collecting and 
reporting outcome data. 

To support states in constructing their Perkins measures, in 2001 OVAE developed and disseminated a 
core indicator handbook, which provides a description of measurement objectives for each core indicator, 
examples of measure construction, definitions of key terminology, a summary of data quality challenges, 
and answers to frequently asked questions. MPR recommends that the NMPED adapt the content of this 
technical assistance handbook to reflect current state measurement approaches, and that the state 
disseminate the handbook statewide to support educators in collecting local data. A copy of the OVAE 
measurement guidebook is included in Appendix B. 


Challenge: Secondary educators have difficulty identifying CTE concentrators. 

Participants reported that in the absence of objective criteria for identifying a CTE course or program 
sequence, they are identifying CTE concentrators using subjective criteria. 
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Recommendation 


The NMPED is in the process of restructuring the state’s vocational program organization around the 16 
career clusters identified by the National Career Technical Education Foundation (www.careerclusters.org). 
As part of their 2005-06 Perkins basic grant application (Due April 29, 2005), district administrators are 
required to identify how their existing CTE coursework fits into the 16 career cluster areas, and the 
programs of study that will apply within each cluster. Specifically, for each program of study, the district 
application must include a description of the academic and CTE courses that must be completed, propose 
an advisory committee for the program of study; identify a CTE organization affiliated with the program of 
study; and summarize the funding provided for CTE by the eligible institution. A CTE student will include 
those completing three or more CTE courses in an identified program or coherent sequence of courses or 
instructional units. 

MPR researchers believe that the NMPED has a unique opportunity to fully automate district identification 
of CTE concentrators — using approved CTE course and program sequence information included in 
districts’ 2005-06 Perkins application — so that educators no longer would be required to enter Perkins data 
into the ADS system. To do so, state ADS programmers would incorporate districts’ course codes and 
pathway sequences into its ADS system. Districts administrators would simply continue to enter into ADS 
the CTE course codes and identifiers of students enrolling in these courses. At year’s end, ADS 
programmers would electronically identify all students who had completed a pathway of 3 or more courses 
in a cluster area — using current and historical district course enrollment data — to select a base of students 
for each district’s Perkins measure. 

While fully automating reporting would introduce some up-front programming costs, once entered, district 
course and program sequence codes would only need to be updated to reflect changes in district CTE 
offerings. One caveat is that, since student identifiers were only recently incorporated into the ADS system, 
the NMPED would only be able to track students from the 1 20 th day of the 2004-05 academic year forward. 
This would likely require that the state develop transitional measures to track short-term student outcomes. 


Challenge: District administrators do not understand how Perkins outcomes are calculated. 

Participants reported that they do not understand how the information they enter into ADS is used by the 
state to calculate their performance outcomes. Many reported that their first exposure to their Perkins 
performance data occurs when it is posted on the state website. Based on participant comments, it also 
appears that few local education agencies are using their Perkins data to undertake program improvement 
efforts, in part because they do not have easy or timely access to their data. 
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Recommendation 


Participants requested that the NMPED develop a more immediate feedback mechanism that would allow 
them to review state compilations of their data prior to its posting on the Department website. It may be 
possible, for example, for the NMPED to provide local administrators with summary data on their program 
performance once ADS data are ‘frozen’ in August of each academic year. State staff could then work with 
district administrators to review information, which would require that NMPED staff manually fix errors in the 
ADS system prior to calculating final state outcomes for posting on the state education website. 

MPR also recommends that the NMPED communicate to district administrators how their data are used in 
ADS to assess student performance on each measure. Demystifying the reporting process can contribute 
to data quality by encouraging local educators to review important data fields for accuracy prior to its 
submission to the state. The NMPED may also wish to encourage district administrators to improve the 
dissemination of CTE concentrators’ academic test results to school staff. Although the state routinely 
provides districts with copies of their academic performance data — disaggregated by school site and 
individual students — participants reported that they do not always have access to this information. 

Given participants’ interest in using data for continuous improvement purposes, NMPED may also wish to 
make available resource materials that assist district staff in using their Perkins data to institute local 
reforms. In particular, MPR recommends that NMPED consider distributing the OVAE guidebook 
“Improving Performance: A Five Step Process." This guide was developed by OVAE to help secondary 
and postsecondary educators use their data to improve their performance on the Perkins core indicators. It 
describes a generic five-step process that can be applied in its existing form or modified to address state 
conditions. A copy of the Quick Reference Guide, outlining the five-step process, is included in Appendix C 
of this report. A copy of the complete publication may be downloaded from the Resource Library section of 
OVAE’s Peer Collaborative Resource Network website (www.edcountabilitv.net) . 


PERKINS MEASURE 1S1: ACADEMIC ATTAINMENT 

Challenge: The New Mexico High School Competency Examination does not provide valid 
information on CTE concentrators’ academic achievement. 

Current state reporting requirements for 1S1 stipulate that students self-identify their CTE status at the time 
they take the New Mexico High School Competency Examination in grade 10. Since there is some 
question whether it is possible for a student to complete a meaningful sequence of three vocational 
courses by the 10 th grade, or whether a student can accurately identify himself or herself as a CTE 
concentrator, participants suggested that the current measure does not provide useful information for 
policymaking purposes. 
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Recommendation 


The NMPED is in the process of adopting the New Mexico Standards Based Assessment (NMSBA) as its 
new measure of academic performance for indicator 1S1. The NMSBA is a criterion-referenced test, 
administered in grades 4, 8, and 1 1 , that is currently used by the state to assess whether school districts 
and schools are making Adequate Yearly Progress in reading and math, as called for in the federal No 
Child Left Behind Act. The state is currently reviewing 11 th grade performance results from the 2003-04 
school year, the first year in which it was administered at the secondary level, prior to its statewide adoption 
as a replacement measure for Perkins. 

Perhaps the simplest way of collecting data on CTE concentrators would be to have students self-report 
their CTE status at the time of NMSBA administration. This would likely undermine the quality of state data, 
however, because students do not always know if they qualify as a CTE concentrator and because many 
students may not have completed three courses in a CTE pathway at the time of test administration. 
Instead, MPR recommends that the state develop procedures for collecting retrospective data on the 
academic achievement of vocational concentrators, either at the time they complete the third course in a 
vocational sequence or at a fixed point in time, such as when they become eligible to enter the base of 
measure 2S1 (i.e., when they begin their enrollment in the 12 th grade). 

To avoid imposing a data burden on districts, the NMPED could use current and historical course taking 
data contained within ADS to automatically identify CTE concentrators, as described above. The state 
could then forward a list of student identifiers to the state-testing agency, which would identify the number 
of students who met or exceeded state performance targets. Although this measurement strategy would 
not provide information on the contribution that CTE concentration has on students’ academic skill 
development, outcomes could be used to assess whether students who complete a CTE concentration are 
as likely as other students to pass the NMSBA. 


PERKINS MEASURE 1S2: VOCATIONAL SKILL ATTAINMENT 
Challenge: Measure construction is not clearly defined. 

The current state measure requires local education agencies to report on the total number of students 
enrolled in CTE courses or programs who passed with a grade of “C” or better. Participants reported 
difficulty in determining whether they should be reporting outcomes for a specific course, or averaging 
across a series of courses within a program area. Since the terms “course” and “program” are used 
interchangeably, participants do not understand the measure as it is currently constructed. Moreover, the 
measure appears to include all students enrolled in CTE coursework, irrespective of whether they are CTE 
concentrators. 
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Recommendation 


In the short term, the NMPED may wish to clarify the construction of its CTE skill attainment measure and 
the population of students included in its base. Since it appears that Congress will no longer permit states 
to report student grades as a measure of CTE skill attainment, many of the issues surrounding the state’s 
use of GPA will disappear with the upcoming reauthorization. 

According to NMPED administrators, the state is currently working with the National Occupational 
Competency Testing Institute (NOCTI) to develop CTE assessments corresponding to the 16 career 
clusters identified by the state. The state is also piloting the use of NOCTI assessments in a number of 
sites, with the expectation of transitioning to these assessments in time to comply with new Perkins 
accountability provisions. 

To ensure the CTE educators are prepared for the new measurement approach, MPR recommends that 
NMPED provide local agencies with sample exams and, where possible, detailed descriptions of the 
standards associated with each cluster area. This can help educators align their CTE course curricula with 
state assessment instruments, a critical first step in ensuring that testing outcomes reflect the knowledge 
and skills that students are taught. 


Challenge: Educators are unable to track outcomes of students who are concurrently enrolled. 

Students in some districts have the option of enrolling in a nearby postsecondary institution to obtain 
advanced CTE training. Participants reported that they are not currently including students who are 
concurrently enrolled in their current measures, since it is not clear whether the secondary or 
postsecondary institution has ownership of these students. Accordingly, participants requested that the 
state clarify whether high school students concurrently enrolled in postsecondary CTE coursework should 
be reported by the high school or postsecondary institution. 

Recommendation 

According to NMPED administrators, current state policy dictates that students enrolled in Dual Credit 
coursework — those providing both high school and college credit — should be reported only by the 
secondary institution. Under new rules issued by the NMPED, secondary students enrolled in Early 
Admission programs — where no high school credit is offered — should be reported as a postsecondary 
student. To assist secondary and postsecondary administrators in classifying students, MPR recommends 
that the NMPE release a memo detailing current and new state policies, and that it add documentation to 
the ADS data manual explaining how student information should be entered. 
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PERKINS MEASURE 2S1: COMPLETION 


Challenge: CTE concentrators who complete their schooling prior to or subsequent to their projected 
graduation date are not counted as completers. 

The construction of the current state measure restricts reporting to 12 th grade CTE concentrators who were 
enrolled at the beginning of the school year. This means that juniors who graduate early and seniors who 
graduate late (i.e., during the summer months) are not counted in the existing measure. 

Recommendation 

To obtain a more accurate count of the number of students who graduate, participants asked that the 
NMPED consider expanding its definition of a CTE completer to include CTE concentrators who graduated 
during their junior year or in the summer following their senior year. To simplify reporting, participants 
suggested that juniors who graduated early be included in the current year’s 12 th grade measurement 
totals, while seniors graduating late be included in the following year’s totals. 

MPR recommends that the state review how ADS data are collected to determine whether there are 
technical constraints that would affect district reporting on this measure. If the timing of district data 
submissions to ADS restricts reporting on some students (e.g., seniors graduating in the summer months), 
then MPR recommends that the state survey a sample of district administrators to assess the scope of this 
problem. If numbers warrant, MPR recommends that the state convene a working group of district CTE 
administrators to consider participant proposed options, a well as other measurement possibilities. For 
example, it may be possible for districts to provide student identifiers to the state of students graduating 
during the summer months, which ADS programmers could use to identify CTE concentrators who might 
otherwise not be counted. 

It is worth noting that under the current system, while juniors graduating early are not included as a district 
success, they also do not enter the denominator for the following year, meaning that districts are not 
penalized for their early departure. Conversely, including seniors in subsequent year reporting totals will 
lead to double-counting these students: students will be counted in the denominator in the current year, 
and then reappear in both the numerator and denominator of the measure in the following year. Here the 
state has three options: it can either (1) count the student in the current year as an unsuccessful outcome 
and exclude them from subsequent counts; (2) count the student as unsuccessful in the current year and a 
success in the subsequent year, or (3) develop follow-up procedures to account for those who complete 
during the summer months. 
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PERKINS MEASURE 2S2: DIPLOMA WITH A CREDENTIAL 


Challenge: District administrators cannot obtain accurate counts of students obtaining certification. 

Participants reported that CTE concentrators are often too young to test for certain types of certification 
while enrolled in high school, and are consequently not counted in this measure. Moreover, the current 
reporting system does not take into account a student’s efforts towards certification; for example, a senior 
who is 17 may be making good progress towards earning a certification in welding, but since testing occurs 
at age 18, he or she is not counted even though he or she may have obtain the requisite skills in his or her 
secondary studies. 

Recommendation 

Proposed Perkins legislation requires that secondary administrators report on students who attain an 
industry-recognized credential or certificate. Given that most students do not receive these awards while 
enrolled at the secondary level, it may be impossible for local education agencies to be held accountable 
for reporting on these students. Participants suggested that the state consider developing a reporting 
category for CTE students who earned a diploma and who are “working towards" an industry recognized 
credential or certification. 

The NMPED is also working to develop standardized state procedures for reporting on students earning a 
CTE skill certificate. As part of the 2005-06 Perkins application process, local educators must submit a list 
of industry certificates that a CTE instructor will either seek or maintain in order to certify the program and 
appropriately prepare students for a credential. As a consequence, all districts should be able to report on 
students who earn a high school diploma in conjunction with a career cluster credential, although in some 
cases these credentials will focus on broad, crosscutting industry content (e.g., computer literacy). 

The state is also planning to supply administrators attending its upcoming career clusters meeting with 
technical assistance materials developed by the National Career Technical Education Foundation. All 
participants will receive information on the career clusters effort, along with a CD listing each of the known 
industry-recognized credentials within the 16 career cluster areas. While it is recognized that not every 
credential or certification that exists in New Mexico may be identified on these lists, it is anticipated that the 
state-provided technical assistance materials and training will allow for greater consistency and accuracy of 
reporting across schools. 

MPR recommends that the NMPED develop technical assistance materials that detail how local agencies 
should collect and report data on students earning CTE credentials, as well as incorporate these 
instructions in the ADS data manual. It is anticipated that NMPED career cluster specialists will also work 
with school superintendents, principals, and teachers to assist them in tailoring their program and course 
materials to reflect the new career cluster frameworks. 
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Perkins Measure 3: Transition to Postsecondary/Employment or the military 

Challenge: The state does not currently have a measure to assess secondary placement. 

The state is currently unable to collect data on students who transition from secondary education to 
advanced training, employment, or the military. Although the NMPED is working to develop a parental 
release form to permit the collection of student social security numbers (SSN), many participants reported 
that they were unsure of state policy on the collection of SSN, and as a consequence, were not attempting 
to collect this information because they believed to do so would be illegal. 

Recommendation 

The NMPED may wish to draft policy guidance and examples of data collection instruments to assist 
school district administrators in understanding what is and is not permissible with respect to collecting 
student SSN. Representatives of the NMPED may also wish to consult with staff at the Florida Department 
of Education, Office of K-20 Education Information and Accountability to identify strategies that the state 
has developed to collect SSN of secondary students. Contact Jay Pfeiffer, Director, at 850-245-0429. 


Perkins Measure 4: Non-Traditional Participation and Completion 

Challenge: District administrators do not understand how to enter data into the ADS system. 

Participants reported that ADS contains two fields for entering data on nontraditional students, but 
provides no explanation on how district staff should calculate numbers for these measures. 

Recommendation 

MPR recommends that the NMPED develop technical assistance materials to assist district 
administrators in identifying students enrolling in and completing non-traditional programs. For 
example, the Florida Department of Education has compiled a list of non-traditional program areas 
and their associated CIP codes that is posted on the state website. Local administrators use this list 
to identify programs that are associated with occupations that are out of gender balance. A copy of 
this list can be downloaded at: http://www.firn.edu/doe/arm/cctcmis/pubs/codes/code main.htm The 
state should also provide directions in the ADS data manual to assist local Perkins staff in entering 
information into the database. 

If the NMPED were to seek to fully automate its data collection, ADS programmers could work with district 
administrators to identify course code numbers associated with non-traditional program areas. Once this 
information was coded into state programming codes, the state could automatically calculate student 
participations and completion in identified courses, assuming that the state also had a field corresponding 
to student sex in the ADS database. 
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Synthesis of Postsecondary Educators’ Comments 


Community colleges and 4-year colleges and universities in New Mexico maintain sophisticated 
management information systems to administer program data and student records, and employ institutional 
researchers to analyze and disseminate this information. As a consequence, most postsecondary 
institutions have the capacity and technical expertise necessary to collect and report detailed information on 
students participating in CTE programs. 

Participants believed that state postsecondary Perkins data are generally reliable across most institutions, 
in part because colleges worked together at the start of the current act to develop a common set of 
statewide indicators. Accountability system development was coordinated by the NMACC, in association 
with NMPED, which convened a group of institutional researchers (a.k.a. the Performance Indicators 
People, or PIP), to develop a consensus reporting system. 

Although postsecondary institutions compile annual data for each Perkins indicator, few actually use this 
information to evaluate campus performance or to initiate program improvement efforts. One reason is that 
postsecondary Perkins data are considered by some to be too highly aggregated to provide policy-relevant 
information. Rather than use information reported to the NMPED, participants reported that they conduct 
their own, institutionally focused analyses of CTE program outcomes, using data contained within campus 
databases to control for program and student characteristics. 

Participants also reported that Perkins accountability data often have little utility because the measures 
used to track student outcomes for some indicators are meaningless. When asked to explain, participants 
reported that existing state measures were designed to accommodate the reporting capacity of all 
postsecondary institutions in the state, including those that do not maintain certain types of data in 
electronic form. In seeking to develop consensus measures, it was felt that the state adopted a reporting 
system that — though effective for federal reporting purposes — is of limited use for local program 
improvement efforts. 


Challenge: Institutional reporting is not comparable across postsecondary sectors. 

Participants reported that not all higher education institutions shared in the development of the state’s 
Perkins measures or reporting methodologies. While institutional researchers from nearly all community 
college institutions participated in measure development, researchers from 2-year branch campuses 
associated with Eastern New Mexico University, New Mexico State University, and the University of New 
Mexico were not involved throughout the entire development process. As a consequence, Perkins 
measures and data collection methodologies are reasonably aligned across campuses within sectors, but 
less comparable across 2-year and 4-year institutions. 
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Recommendation 


As the state transitions the new Perkins reporting requirements, the NMPED may wish to convene 
representatives of community college and 4-year college and university branch systems to ensure that all 
agencies and sectors are represented in the development of new measures and reporting methodologies. 
The state may also wish to produce better documentation of postsecondary measures and reporting 
methodologies so that institutions unable to participate receive the guidance they need to improve the 
validity and reliability of their data. See the discussion on page 3 of this memo and the materials included in 
Appendix B for MPR’s recommendations for improving state communications regarding Perkins. 


Challenge: Institutions are using different methodologies to collect Perkins data. 

In an effort to standardize Perkins reporting, postsecondary institutional researchers have attempted to 
develop common definitions and reporting methodologies for CTE populations and program coursework. 
For example, to standardize the identification of vocational coursework, the New Mexico Commission on 
Higher Education (NMCHE) convened a Technical Advisory Group to create a standardized list of 
vocational courses, designated by CIP code, for institutions to use for Perkins accountability purposes. 
Participants in the PIP also identified common data collection methodologies to guide institutional reporting. 

Although common reporting frameworks exist, not all institutions appear to be using them to structure their 
reporting. In some cases institutions are unable to apply consensus approaches because the organization 
of facility databases do not conform to suggested guidelines. In particular, some 2-year branch campuses 
are unable to collect some types of data because their databases are modeled on their 4-year college or 
university’s information system, which is structurally different than that of community colleges. It also 
appears that not all campus administrators are using suggested guidelines to collect Perkins data. 

Recommendation 

While differences in institutional data systems may make it impossible to align inter-sector measures, it may 
be possible for postsecondary data administrators to qualify where differences exist and, to the extent 
possible, reduce data error so that all institutions are providing more consistent data. Although it may be too 
late to rectify inter-institutional differences for the current year, the state may wish to work with the NMCHE 
and the NMACC to anticipate and resolve these reporting inconsistencies prior to the introduction of new 
Perkins legislation. The state may also wish to develop a postsecondary measurement guidebook, similar 
to that described in the secondary recommendations, to instruct institutional staff on the procedures for 
collecting and reporting Perkins data. 
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Challenge: On-line reporting instruments are not sufficiently documented or updated. 

To support postsecondary educators in reporting Perkins performance data, in 2003 the state introduced a 
web-based reporting template that allows institutional staff to key enter data directly into state Perkins 
reporting instruments Participants observed that new data entry instructions in the on-line reporting 
template were less detailed than those contained in a previous e-mail version, leading some to suggest that 
postsecondary administrators without prior year experience might encounter difficulty using the template. 
Participants also noted that the data entry template has not been regularly updated to account for academic 
year changes (e.g., 2003 cohort should be changed to the 2004 cohort) 

Recommendation 

The state may wish to consider reviewing existing reporting templates and, where necessary, adding more 
detailed instructions to assist local providers in entering student and program data. One suggestion was 
that the state incorporate instructions from the e-mailed data collection form into the web-based template, 
and that these instructions should be field-tested prior to statewide distribution. State administrators should 
also ensure that data entry instructions are reviewed and updated on an annual basis. 


Challenge: Postsecondary institutions cannot collect accurate data on special population students 

Due to privacy issues, postsecondary administrators are hesitant to collect data for many categories of 
special population students, including displaced homemakers, single parents, and economically 
disadvantaged individuals. As a consequence, administrators either report on individuals for whom they are 
able to infer a special population status or enter a zero in the reporting field. Since entering a zero suggests 
that institutions are not serving these students, administrators requested that NMPED change the reporting 
requirements to permit administrators to enter non-numeric information, such as “Unable to Report.” 

Respondents also requested that the state agency communicate to the US Department of Education the 
challenges that the state faces in collecting data on postsecondary special population students. 

Recommendation 

Although the NMPED has attempted to lobby both the state’s congressional delegation and OVAE 
administrators, it appears likely that Congress will reauthorize Perkins without changing the requirement 
that state and local education agencies report on the performance of special population students. As such, 
the state may wish to take steps to standardize reporting across higher education institutions so that it does 
not appear that some agencies are failing to provide equitable services to some categories of students. 

MPR also recommends that the NMPED convene a working group of representatives from the state higher 
education system — following adoption of new Perkins legislation — to draft standardized procedures for 
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identifying and reporting data on special population students. This may include defining accepted proxies 
for students for whom institutions are currently unable to collect data. The NMPED may also wish to clarify 
state guidelines for identifying students in special population groups; according to conference participants, 
some postsecondary administrators are refusing to collect information in the mistaken belief that it is 
against state law to request certain types of information. 

To improve the validity and reliability of institutional data, NMPED may also wish to consider developing a 
voluntary, standardized, institutionally administered special populations survey that all entering students 
would be asked to complete as part of their application process. Assuming such a survey was legally 
permissible, the use of a standardized data collection instrument would remove the subjectivity associated 
with current student classifications methods, while ensuring some degree of consistency across institutions. 
An example of a student survey used by Lewis and Clark Community College in Illinois is included in 
Appendix D of this memo. 


Challenge: Postsecondary Institutions are not receiving credit for all vocational concentrators who 
complete their intended program of studies 

The state’s current definition of a postsecondary vocational concentrator is a student who completes nine or 
more CTE credit hours in a vocational program area and who declares a CTE major. This definition 
penalizes institutions because many students who meet the course-taking threshold subsequently declare 
a CTE major to qualify for financial aid, even if they have no plans to complete a degree or credential. This 
means that students can enter the denominator of many of the state’s postsecondary measures even 
though they have plans that prevent them from entering the numerator. Moreover, students who complete 
certain types of coursework (e.g., skill upgrading for career advancement) may not be counted as a 
success even though they have fulfilled their enrollment objective. 

Recommendation 

It is likely that new 2005 Perkins accountability provisions will require that states report on students who 
attain an industry-recognized credential, certificate, or degree, as well as who transfer to a baccalaureate 
degree-granting program. While this expanded definition will enable institutions to receive credit for 
students completing certificate or credential programs, institutions will still face challenges counting 
students who complete a program or series of courses that fulfill their enrollment objective (e.g., skill 
upgrading), but who have no intention of graduating. 

To improve reporting accuracy, the state may wish to work with postsecondary educators to develop criteria 
for differentiating students who are actively pursuing a degree, credential, certificate, or transfer from those 
attending for other reasons (e.g., skill upgrading or personal development). One participant also suggested 
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that the state consider making available a state licensing databases to permit institutions to conduct 
matches on completers who earn a state recognized license after exiting a program. 

Since institutions may also face difficulties differentiating students who “stop out” from those who transfer to 
an out-of-state 4-year college, the state may wish to develop postsecondary data sharing agreements with 
neighboring states (see below) or consider joining the National Student Clearinghouse, located in Herndon, 
Virginia. Clearinghouse data are used to verify enrollments of student loan recipients who defer pay back 
obligations, and the database includes information on approximately 85% of the nation's postsecondary 
enrollments. Since this is a fee-based service, MPR recommends that the NMPED consult with the 
NMCHE to assess the scope of out-of-state transfer at the postsecondary level, as well as other potential 
benefits of tracking students — extending beyond Perkins — before subscribing to this service. 


Challenge: Postsecondary institutional researchers have difficulty tracking students into 
employment or advanced education 

Some participants noted that they are prohibited from gaining access to student social security numbers 
because institutional administrators do not always understand Federal Education Rights and Privacy Act 
(FERPA) requirements. Operating under the mistaken belief that they cannot release personally identifiable 
information, a small number of institutional administrators are undermining institutional capacity to 
participate in post-program follow-up using Ul wage record data. 

Recommendation 

The NMPED may wish to produce a memorandum to secondary and postsecondary administrators 
clarifying what types of student information are and are not permissible to be released under FERPA and 
state regulations. 


Challenge: Postsecondary institutional researchers are unable to track placement and employment 
outcomes for students who leave the state or enter the military or federal employment 

Community colleges located near state borders often serve students who are likely to find out-of-state 
employment, reducing institutional placement counts. Moreover, all institutions within the state are 
hampered in tracking students who enter federal employment or the military. 

Recommendation 

The New Mexico Department of Labor (NMDOL) has been working to negotiate Ul wage record sharing 
agreements with neighboring states to improve the tracking of students across state lines. Accordingly, 
MPR recommends that the NMPED follow-up with NMDOL staff to assess the status of state negotiations 
and their implications for Perkins. In the event this effort is not moving forward, NMPED might wish to 
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consult with other states that have developed these types of record-sharing agreements. For example, 
Ohio has a multi-state agreement with several bordering states governing the exchange of Ul wage record 
data. Contact Douglas Holmes, Department of Job and Family Service at 614-644-9178 to obtain a copy of 
the Memorandum of Understanding and contractual agreements underlying this exchange effort. 

It was also suggested that the state explore the potential for conducting administrative record matching with 
federal employment and military databases to capture outcomes for students who are employed but not 
reported in state Ul wage records databases. See Appendix E for an example of a data sharing agreement 
between the Illinois State Board of Education and the U.S. Office of Personnel Management. The NMPED 
may also wish to consider establishing a data sharing agreement with the U.S. military to identify students 
who enlist. Contact Mike Dove of the Defense Manpower Data Center, DOD Center Monterey Bay, 400 
Gigling Road, Seaside CA 93955-6771 or e-mail the center at: webmaster@osd.pentaqon.mil 


Challenge: Postsecondary institutions are often unable to identify Tech Prep students 

Participants reported difficulty distinguishing Tech Prep students from non-Tech Prep students who are 
participating in postsecondary CTE courses articulated with secondary vocational programs. Moreover, 
since secondary students do not always know that they are classified as a Tech Prep student, matriculates 
are generally not able to self-identify themselves upon entering a community college. 

Recommendation 

The state has a number of options to improve the identification of Tech Prep students. 

Option 1: Maintain a Centralized Tech Prep Database 

Secondary institutions could forward identifiers of students participating in secondary Tech Prep programs 
to the state or the NMCHE, which would maintain a Tech Prep database. These identifiers could include 
students’ SSN or unique identifying characteristics, such as students’ first name, last name, birth date, and 
school of attendance. (Note: see Appendix F for a memo on the legality of transferring student identifiers 
from the Illinois Board of Education). Postsecondary institutions could then either receive a file containing 
this information, against which they would conduct their own administrative record matches, or forward a file 
to the state (or NMCHE) containing identifiers of students participating in postsecondary Tech Prep 
programs. This would support tracking students who transition from a secondary to a postsecondary Tech 
Prep program for which a specific pathway exists, as well as for students who enroll in Tech Prep 
coursework at a non-affiliated community college. One drawback of this approach is that it requires the 
state or the NMCHE to maintain and administer a Tech Prep database, which may be resource intensive. 
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Option 2: Conduct Retrospective Matching 

The state may wish to work with secondary and/or postsecondary educators to standardize procedures for 
tracking Tech Prep students as they transition across sectors. One approach, currently used in California, 
would be for postsecondary institutional researchers to identify students participating in postsecondary CTE 
programs who were graduates of a high school with an affiliated Tech Prep pathway. Students enrolled in 
these postsecondary programs — either one or two years after graduating from their feeder high school — 
would automatically be classified as a Tech Prep student. 

To reduce the probability of reporting false positives (i.e., students who attended a feeder high school but 
who were not enrolled in a Tech Prep pathway), postsecondary institutions could share a list of their 
identified Tech Prep students with their feeder high schools, who would confirm or deny student 
participation. See Appendix G for a copy of California’s State Plan for Vocational and Technical Education, 
which details the state’s tracking approach. 

Option 3: Coordinate Secondary and Postsecondary Reporting 

A third option would be to require secondary educators to forward the identifiers of secondary Tech Prep 
students to their affiliated postsecondary institutions. Postsecondary institutional researchers would then 
attempt to match secondary student identifiers with those of postsecondary students enrolled in a 
recognized Tech Prep pathway, either using students’ SSN or by performing probabilistic matching using a 
combination of student identifiers (e.g., name, birthday, school). 

Option 4: Provide Tech Prep Students with a Portable Transcript 

Rather than rely on secondary or postsecondary administrators to identify Tech Prep students, Nevada 
relies on students to identify themselves by attaching fiscal incentives to completion of a secondary Tech 
Prep program. Nevada high school students who complete an approved Tech Prep sequence with a grade 
of an “A” or “B” can apply for a specialized high school transcript that, when presented at a participating 
community college, automatically provides them with college credit. Colleges use these transcripts as a 
way of identifying Tech Prep students for postsecondary Perkins follow-up. While the system is not 
perfect — it is estimated that only 50 percent of eligible students in Nevada apply for a Tech Prep 
transcript — some postsecondary schools report that as many as 90 percent of students entering eligible 
coursework present a secondary transcript. For more information on the Nevada model, contact John 
Bearce of the Community College of Southern Nevada by phone at 702-651-7454 or by e-mail: 
iohn bearce@ccsn.edu 

State administrators may also wish to access the Tech Prep Compendium maintained on the website of the 

Office of Community College Research and Leadership, affiliated with the College of Education at the 

University of Illinois at Urbana-Champaign, for additional resources on identifying and tracking Tech Prep 

students across secondary and postsecondary sectors: http://occrl.ed.uiuc.edu/ 
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Appendix A 



New Mexico Public Education Department 
Perkins Accountability Technical Assistance Workshop 


9:00 am 

9:15 -10:15 am 

10:15-10:30 am 
10:30- 11:30 am 

11:30- 1:00 pm 
1:00 - 2:30 pm 

2:45-3:00 pm 


Tuesday, April 5, 2005 


Welcome and Introductions 


Overview of Federal Perkins Accountability System 

Summary of federal reporting requirements and the challenges that state 
and local education agencies face in collecting data. 


Break 


Collecting and Reporting Accurate Data 

Overview of the key components of a quality data collection system, 
followed by secondary and postsecondary breakout discussions to 
identify issues affecting data quality. 


Lunch 


Brainstorming Future Directions 

Participants will meet in secondary and postsecondary breakouts to 
identify challenges, recommendations, and technical assistance needs for 
each Perkins indicator. Group members will compile the challenges they 
face in reporting quality data and prioritize their most pressing technical 
assistance needs. 


Where Do We Go From Here? 

Discussion of how meeting results will be used to support future state 
initiatives and directions. 


3:00 pm 


Adjourn 




Core Indicator Work Session: 
Improving Perkins Data Quality 


1:00- 1:10 pm 

Overview of the Exercise 

Facilitators will review the purpose of the exercise and work with groups 
to select recorders. 

1:10-1:40 pm 

Review First Indicator 

Group members will answer the following questions about their assigned 
indicator, using the following suggested timetable: 


• 10 minutes - What are the challenges your education agency 

faces in collecting data for this indicator? 

• 10 minutes - How would you improve the collection of data 

for this measure, or if necessary the measure 
itself, to improve its accuracy and utility? 

• 10 minutes - What kinds of state technical assistance would 

you need to collect quality data on this measure? 


Each group will write their responses on flip charts, recognizing that 
these charts will be collected and used by the state to support system 
development. 

1:40 -2:10 pm 

Report-Out 

Group members will report-out on the challenges, recommendations, 
and technical assistance needs identified for each indicator and collect 
feedback from the larger group. Group recorders will provide copies of 
written charts to their secondary or postsecondary facilitator. 

2:10 - 2:30 pm 

Brainstorming Future Directions 

Group members will compile the challenges they face in reporting 
quality data and prioritize their most pressing technical assistance needs. 

2:30-2:45 pm 

Break and Reconvene 
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Core Indicator I: Student Attainment 


MEASUREMENT OBJECTIVE 
Academic Attainment 

All students who reach a state-defined threshold level of vocational education 
should also achieve mastery of academic knowledge and skills that meet state aca- 
demic standards. To assess attainment of academic knowledge, Congress requires 
state and local education agencies to report on the academic outcomes of students 
who have concentrated in vocational education and left secondary education or 
stopped program participation at the postsecondary level. 

Vocational and Technical Skill Attainment 

All students who reach a state-defined threshold level of vocational education 
should master knowledge and skills that meet state-defined and industry-validated 
career and technical skill standards. To assess whether vocational concentrators have 
attained vocational and technical skills, state and local education agencies should 
report on the vocational outcomes of students who have left secondary education or 
stopped program participation at the postsecondary level. 


MEASURE CONSTRUCTION 

Academic and vocational attainment rates are based on the number of vocational 
education concentrators who have left secondary education or stopped postsec- 
ondary program participation and mastered state-established academic knowledge 
and skills and mastered career and technical skills. Rates are calculated using the fol- 
lowing formulas: 

Secondary Academic Attainment 

Numerator: Number of students reaching a state-defined threshold level of 
vocational education who have met state academic standards and have left 
secondary education in the reporting year. 
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Core Indicator I: Student Attainment — continued 


Denominator: Number of students reaching a state-defined threshold level 
of vocational education and who have left secondary education in the 
reporting year. 

Secondary Vocational and Technical Skill Attainment 

Numerator: Number of students reaching a state-defined threshold level of 
vocational education who have met state-established, industry-validated 
career and technical skill standards and have left secondary education in the 
reporting year. 

Denominator: Number of students reaching a state-defined threshold level 
of vocational education and who have left secondary education in the 
reporting year. 


Postsecondaiy Academic Attainment 

Numerator: Number of students reaching a state-defined threshold level of 
vocational education who complete a postsecondary program and who have 
met program-defined academic standards and have stopped program partic- 
ipation in the reporting year. 

Denominator: Number of students reaching a state-defined threshold level 
of vocational education who complete a postsecondary program and who 
have stopped program participation in the reporting year. 

Postsecondaiy Vocational and Technical Skill Attainment 

Numerator: Number of students reaching a state-defined threshold level of 
vocational education who complete a postsecondary program and who have 
met program-defined and industry-validated career and technical skill stan- 
dards and have stopped program participation in the reporting year. 

Denominator: Number of students reaching a state-defined threshold level 
of vocational education who complete a postsecondary program and who 
have stopped program participation in the reporting year. 
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Core Indicator I: Student Attainment — continued 


DEFINITIONS 

Vocational Concentrator 

A student who enrolled in a threshold level of vocational education. 

Threshold Level of Vocational Education 

A program/sequence of courses or instructional units that provides an indi- 
vidual with the academic and technical knowledge/ skills/proficiencies to 
prepare the individual for employment and / or further or advanced educa- 
tion. 


Left Secondaiy Education 

Either graduating from high school or withdrawing from or ceasing to 
attend high school for a state-defined period of time. 

Stopped Program Participation 

Completing a program, withdrawing from a program, or not enrolling in 
program courses for a state-defined period of time. 


DATA COLLECTION STRATEGIES 

States are required to report academic attainment outcomes and career and techni- 
cal skill attainment outcomes on all vocational concentrators who leave high 
school/stop participation in a postsecondary program. States typically use state or 
national academic assessments, grade-point average, or course or program comple- 
tion as approaches to collecting data on these students. 
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Core Indicator I: Student Attainment — continued 


STRATEGIES FOR IMPROVING DATA QUALITY 

Challenge: State-level assessments are used to measure academic performance in my 
state, but we are not getting back test scores for many of the students whom we expect to 
take the exams. What can be done to collect this data ? 

Solutions: 

• Track and keep a record of the vocational concentrator students whom you 
expect to take the assessment and compare your list to the list of data scores 
that you get back. 

• Establish a follow-up procedure to pinpoint reasons for non-matches. For 
example, is there a problem with the identifiers and name fields that can be 
corrected? Did students who were expected to take the exam not take it for 
reasons of illness or refusal? Once the source of the non-match is estab- 
lished, it may be clearer what steps need to be taken to gather the test 
scores. 

• Evaluate whether the proportion of vocational concentrators who took the 
assessment in your school or college is significantly different from that in 
other schools. This process might help in establishing whether there is a 
system-wide problem or whether it may be unique to your school. 

Challenge: My state uses assessments to measure academic attainment, but a large num- 
ber of students in my LEA are exempt from testing. How should I collect data on these 
students ? 

Solutions: 

• Identify vocational concentrators not taking assessments to evaluate the 
extent to which the full population is not being included in the measure- 
ment. You should be reporting on at least 80 percent of vocational concen- 
trators. 
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Core Indicator I: Student Attainment — continued 


• Make sure that exempt students have an Individual Education Plan in place 
that includes alternative assessments of skills that measure academic attain- 
ment that will meet state standards. 

Challenge: My state uses national! state assessments to measure vocational attainment. 

However, some of the programs offered by my college are unique to my area and not list- 
ed under state assessments. What can I do to collect data for students in these programs? 

Solutions: 

• Work with your state to determine the best way to develop local assess- 
ments for these programs that the state would recognize and count as meet- 
ing state-established criteria. 

Challenge: What is the standard for determining completion in a course or program? 

Solutions: 

• Review your states requirements for which courses are included in academ- 
ic attainment and which courses are included in technical skill attainment 
and the associated standards for “passing” (i.e., the grade threshold). 

• Only those vocational concentrators who complete the state-required pro- 
gram courses should be counted in the numerator. 

Challenge: How do I know which courses to include in the calculation of GPA? 

Solutions: 

• To identify the courses in your LEA that should be considered as academic 
requirements and the courses that should be considered as vocational 
requirements, review your state plan for the criteria that has been agreed 
upon statewide. 
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Core Indicator I: Student Attainment — continued 


Challenge: Our attainment data suggests that our vocational concentrators’ performance 
is not as high as others in our LEA and across the state. How can we ensure that our 
data are accurate ? 

Solutions: 

• Review the data to make sure that only vocational concentrators are includ- 
ed in the numerator and denominator and that you have adequate coverage 
of your vocational concentrators (at least 80 percent). It could be that some 
students have been improperly identified as vocational concentrators and 
are lowering outcomes. Check for missing data. 

• Check to make sure that the data are being accurately compiled, entered, 
and tabulated. Provide staff training on the correct procedures for entering 
and extracting the data and on the appropriate fields to include in calcula- 
tions. 

• Compare your timing for data collection with other LEAs. If you are col- 
lecting data at different times or inconsistently, your results may differ from 
other schools over time and across populations. 

Challenge: My state is using program completion for both postsecondary attainment 
measures. The state sent back my numbers to verify their accuracy, but the number of 
students being counted appears low to me. What can I do? 

Solutions: 

• Check on whether your interpretation of the student population in the 
denominator agrees with the population that the state measure specifies. 
Make sure that the student population that you are counting matches with 
the state. 

• Verify that the degree and certificate programs that you are counting for the 
population correspond with the programs that the state is counting. The 
state may not be counting all of the programs that you are including. 
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Core Indicator I: Student Attainment — continued 


It is important to make sure that the time period for which you are report- 
ing is correct. Check that you and the state are reporting on students over 
the same period of time. 

Re-run your numbers and compare them to the state’s to see if changing 
any of your interpretations of the measure has an impact. 


Challenge: Our state is using course completion/program completion to measure voca- 
tional skill attainment. I am concerned that we are not accurately measuring the skills 
that matter to our employers. How can we ensure that completion equates with compe- 
tency and the attainment of skills? 

Solutions: 

• Check to make sure that all programs and courses reporting under this 
measure have identified skills. It is important that all educators are aware of 
the skills that must be attained to count as completing a course. 

• Establish common practice in a vocational area. For example, achievement 
of 80 percent of identified course competencies equates with completion. 

• Build educators’ capacity to conduct course testing and assessments to 
improve the validity and reliability of the assessments. Be sure that voca- 
tional competencies being measured in classroom assessments and tests 
align with course content. 

• Structured response assessments may face problems of validity, while alter- 
native assessments can be unreliable. Mixing assessment modes may 
improve the measurement of competency in a vocational area. 
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Core Indicator I: Student Attainment — continued 


FREQUENTLY ASKED QUESTIONS 

Q; What can I do to improve on my performance for attainment ? 

A: Differentiate between data quality and program improvement. Verify that 
the data are accurately reflecting student attainment and implement strate- 
gies to improve data quality. Evaluate patterns and trends in the data and 
identify root causes for performance: are poor performers all in one pro- 
gram area? Do lower-performing students share characteristics or barriers 
that can be addressed? Are high-performing students enrolling in different 
courses than the general population? After identifying possible causes for 
performance, work with educators to help them understand the data and 
develop ways to improve programs so student competency and achievement 
rises. Technical assistance, professional development, and capacity-building 
efforts will promote buy-in and eventual improvement in performance. 

Q; How should a vocational concentrator who transfers from another system be reported ? 

A: LEAs should take responsibility only for students who are enrolled in their 
institution. Students transferring into your LEA become a part of your sys- 
tem and should be included in your measurement systems. Students trans- 
ferring out of your system should be subtracted from the numerator and 
denominator of all applicable measures. Note that postsecondary students 
who transfer out of your LEA to a 4-year institution may be counted as a 
placement, depending how your state has structured its measure. 

Q: How should academic outcomes be reported if the state exam contains multiple assess- 
ments in different content areas ? 

A: States should have developed a combined measure — the percentage of con- 
centrators who meet standards (i.e. pass exams) in all content areas. If states 
cannot develop a combined measure, then OVAE will work with them to 
negotiate performance levels for each content area. Work with your state to 
discuss reporting for a combined measure if it has been developed or 
reporting on the different content areas if a combined measure has not 
been developed. 
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Core Indicator I: Student Attainment — continued 


Q: How should I count students who take an assessment prior to becoming a vocational 
concentrator ? 

A: If your state chooses to use a standardized state exam administered early in 
a student’s secondary program, it may be deemed the best measure available 
to the state. In this case, a state should consider setting performance targets 
that are similar to those for all students, with the intent of showing that 
students concentrating in vocational programs enter programs at or above 
the academic skill level of other students. 

Q: Students in my state have the opportunity to take the state assessment over several 
years. How do I know when to collect data on students ? 

A: Consult with your state vocational administrative team to determine the 
best measure for assessing student performance. 
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Core Indicator 2: Completion 


MEASUREMENT OBJECTIVE 

All students who reach a threshold level of vocational coursework should go on to 
complete their academic and vocational studies and graduate. To assess student suc- 
cess in attaining their educational goals, Congress requires state and local education 
agencies to report completion rates for secondary and postsecondary students who 
have achieved a minimum level of vocational coursework. States offering proficien- 
cy credentials in conjunction with a secondary school diploma are also expected to 
report student-credentialing rates for years 3-5 of the Perkins Act. 


MEASURE CONSTRUCTION 

Completion rates are based on the number of students who enroll in or complete a 
state-defined threshold level of vocational studies. Rates are calculated using the fol- 
lowing formula: 

Secondaiy Completion 

Numerator: Number of students reaching the state-defined threshold level 
of vocational education who have attained a high school diploma or its rec- 
ognized state equivalent and who have left secondary education in the 
reporting year. 

Denominator: Number of students reaching the state-defined threshold 
level of vocational education and who have left secondary education in the 
reporting year 
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Core Indicator 2: Completion — continued 


Secondary Proficiency Credential With Secondary Diploma 

Numerator: Number of students reaching the state-defined threshold level 
of vocational education who have attained a proficiency credential in con- 
junction with a high school diploma or its recognized state equivalent and 
who have left secondary education in the reporting year. 

Denominator: Number of students reaching the state-defined threshold 
level of vocational education who have received a high school diploma or 
its recognized state equivalent and who have left secondary education in the 
reporting year 


Postsecondaiy Completion 

Numerator: Number of students reaching the state-defined threshold level 
of vocational education who receive or were eligible to receive a postsec- 
ondary degree, certificate, or credential and who stopped program partici- 
pation in the reporting year. 

Denominator: Number of students reaching the state-defined threshold 
level of vocational education who are not yet eligible to complete plus those 
students who received or were eligible to receive a postsecondary degree, 
certificate, or credential, and who stopped program participation in the 
reporting year. 


DEFINITIONS 

Equivalency Certificate 

A state-recognized award that signals that the recipient has completed an 
approved program of studies at the secondary level. These awards may 
include a Certificate of Competence, Certificate of Completion or GED 
certification. Due to the difficulties of counting GED students who are eli- 
gible to graduate, most states are choosing not to include GED students in 
their measure. 
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Core Indicator 2: Completion — continued 


Proficiency Credential 

A state-developed credential that certifies that its recipient has attained a set 
of recognized knowledge and skills associated with a vocational program or 
career area. Proficiency credentials are typically awarded in conjunction 
with a secondary school diploma. 


DATA COLLECTION STRATEGIES 

States are required to report counts of the number of students who complete a 
threshold level of vocational coursework and who go on to graduate from their sec- 
ondary school or postsecondary institution. Data are typically collected using stu- 
dent transcripts, with collection occurring as of a state-specified cutoff date or time 
period that may differ for secondary and postsecondary institutions. 


STRATEGIES FOR IMPROVING DATA QUALITY 

Challenge: Completion rates for vocational concentrators in my LEA are much lower 
than the state average; however, my LEA graduation rate is actually quite high. Why the 
difference ? 

Solutions: 

• Assess whether vocational concentrators are actually more likely than other 
students to drop out or stop out prior to graduation. It may be that higher 
completion rates of other students are masking lower performance among 
vocational concentrators. If so, develop intervention strategies to improve 
completion rates for vocational concentrators. 


Core Indicator 2 | Page 3 


Core Indicator 2: Completion — continued 


• Determine whether you are using the state-recognized definition for voca- 
tional concentrators and completers. It may be that you are collecting data 
on a different student population than other sites. 

• Check whether you are missing completion data on students who, once 
classified as vocational concentrators, are overlooked for graduation. This 
may occur for a variety of reasons, including data entry errors, early or late 
graduation (e.g., following summer school), or outstanding fines or debts 
that must be cleared prior to diploma or degree award. 

• Completion rates may vary with program type or size. Check to see if the 
mix of program offerings at your LEA is similar to those in other sites. 

• Assess whether the base population of students has changed over time. 
Students classified as vocational concentrators who subsequently transfer to 
other schools or institutions should be subtracted from the base of students 
to ensure that only students who are eligible to complete are included in 
the denominator. 

• At the postsecondary level, assess whether low rates can be traced to stu- 
dents who shift from full-time to part-time status or who stop out late in 
their program. It may be efficient to target resources on students who are 
relatively advanced in their program but who are at high risk of changing 
their enrollment status. 

Challenge: Completion data reported by different sites in my LEA, or within a single 
site, vary over time. What can I do to improve the reliability of reporting? 

Solutions: 

• Determine whether school or institutional staff responsible for collecting 
the data fully understand reporting parameters and the procedures required 
to locate information in student files. Check to see if other fields may con- 
tain information that will help determine student outcomes. 
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Core Indicator 2: Completion — continued 


• Schedule annual training sessions with staff from your LEA who are 
responsible for collecting completion data to ensure all follow similar data 
collection procedures. 

• Consider developing written, step-by-step instructions, tailored to the data 
systems used in your state, to assist LEA staff in collecting information. 

• Check to ensure that all LEAs and sites within your LEA are collecting data 
according to the state-defined period. Sites that put off data collection until 
the end of the summer may report higher completion rates than those that 
record data at the end of the academic year (i.e., June versus August collec- 
tion). 

Challenge: LEA staff do not take the reporting requirements seriously. How can I 
encourage individuals to collect accurate data ? 

Solutions: 

• Create a statewide working group of influential educators and management 
information specialists to identify the obstacles to the collection of high 
quality, easily accessed data. Based on feedback, develop procedures to sim- 
plify data collection. 

• Educate LEA staff on the potential uses of accurate student completion 
data and develop training materials that will enable them to use the data 
they collect to improve program provision. 

• Remind LEA staff that sites failing to make progress toward state perform- 
ance levels may be required to develop local program improvement plans. 

• Publicize completion rates to allow students, parents, and the community 
to see the results that are being reported to state and federal policymakers. 
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Core Indicator 2: Completion — continued 


Challenge: The measure for completion in my state has changed and/or state policymak- 
ers are unable or unwilling to designate a specific measurement strategy. What should I 
do ? 


Solutions: 

• Consistency is the most important part of any measurement system. If 
measures change over time, consider collecting data using early measure- 
ment strategies to assess changes over time. Alternatively, review prior year 
data to identify baseline performance levels using the new measurement 
strategy. 

• Consult with state vocational administrators to identify a consistent meas- 
urement strategy that will allow local agencies to report useful information. 

• Educate state policymakers and LEA staff on the importance of collecting 
stable completion measures. Work with key stakeholders to develop a con- 
sistent approach that will produce useful information for federal accounta- 
bility and local program improvement efforts. 
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Core Indicator 2: Completion — continued 


FREQUENTLY ASKED QUESTIONS 

Q; At what date should data on secondary student completion be collected ? 

A: Many states are using the end of June of a secondary vocational concentra- 
tors senior year as the cutoff date for recording completion. Others are 
using August of the senior year to allow students who enroll in summer 
school additional time to graduate. Consult with your state vocational 
administrative staff to identify the cutoff date that the LEA should use 
when calculating student completion rates. 

Q: My state does not offer a proficiency credential. Must I report on this measure ? 

A: Local secondary agencies in states that do not award proficiency credentials 
are not required to report on this measure. State vocational administrators 
will inform OVAE about the type of diplomas and degrees offered — states 
not awarding this type of credential are exempt from this measure. 
Secondary educators in states offering proficiency credentials are required 
to collect data on student attainment of these credentials; consult your state 
vocational administrative staff or state reporting guidelines to clarify the 
manner in which these data should be collected. Note that postsecondary 
agencies are expected to report on students attaining a postsecondary degree 
or credential awarded by the institution. 

Q: What constitutes postsecondary completion ? That is, must a student graduate to be 

counted as having completed ? 

A: States are using a variety of approaches to document postsecondary comple- 
tion. Some states are basing their calculation on only those students who 
receive a certificate, credential, or Associates degree, while others are also 
including students who are eligible to receive a certificate or degree, irre- 
spective of whether they actually receive the document. A small number of 
states are also including students who transfer to advanced postsecondary 
education, so long as these students complete a rigorous, state-approved 
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Core Indicator 2: Completion — continued 


program that specifies a predefined completion point. Consult with your 
state administrative staff to identify the proper procedure for calculating 
completion rates. 

Q; When should postsecondary institutions assess completion rates, since students may 
enroll indefinitely or stop out prior to returning and completing their degree ? 

A: States have been encouraged to report on vocational concentrators who 

complete their postsecondary program or stop out within a given time peri- 
od designated by the state. In some cases, states are using 11/2 normative 
time as the cutoff for assessing students who attain the vocational concen- 
trator threshold and subsequently complete or stop out from a program. In 
other cases, states may track all vocational concentrators for as long as they 
remain enrolled, meaning that students may be counted in multiple report- 
ing years. Consult your state vocational administrative staff to determine 
the proper procedure for calculating completion rates. 

Q: How can postsecondary institutions report on completion when, in many cases, it is 
not possible to identify a program exiter until one year following program participation ? 

A: Institutions will need to report annual data on student completion rates 
irrespective of the manner in which students are identified. In the event a 
state identifies program exiters using retrospective data, institutions will 
likely report on students who have completed in the prior year, as well as 
those who have not reenrolled during a state-defined period of time. This 
may mean that institutional reporting is lagged one year. 
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Core Indicator 3: Placement 


MEASUREMENT OBJECTIVE 

All students who complete a secondary vocational education program should obtain 
skills that will prepare them for successful transition to postsecondary education or 
advanced training, employment, and / or military service. To assess the quality of stu- 
dent preparation, Congress requires state and local education agencies to report on 
the postsecondary outcomes of students who complete a secondary vocational pro- 
gram and graduate from high school, and at the postsecondary level, who complete 
a postsecondary program in the reporting year. 


MEASURE CONSTRUCTION 

Placement rates are based on the number of high school graduates or postsecondary 
students who successfully transition within a state-defined time period, typically 
between six months and one year of school completion. Rates are calculated using 
the following formula: 

Secondaiy 

Numerator: Number of students who completed secondary vocational pro- 
grams, received a high school diploma or its recognized state equivalent, left 
secondary education in the reporting year, and who were placed in postsec- 
ondary education or advanced training, employment, and / or military serv- 
ice within an designated time period. 

Denominator: Number of students who completed secondary vocational 
education programs and who received a high school diploma or its recog- 
nized state equivalent and left secondary education in the reporting year. 
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Core Indicator 3: Placement — continued 


Postsecondaiy 

Numerator: Number of students who completed a postsecondary program 
in the reporting year and who were placed in further postsecondary educa- 
tion or advanced training, employment, and/or military service within a 
designated time period after stopping participation in the postsecondary 
program. 

Denominator: Number of students who completed a postsecondary pro- 
gram in the reporting year. 


DEFINITIONS 

Postsecondaiy Education or Advanced Training 

A secondary student is considered pursuing higher education or advanced 
job training if he or she enrolls in a public or private postsecondary institu- 
tion, proprietary school, or adult education program during the reference 
period. At the postsecondary level, a student is considered pursuing high 
education or advanced job training if he or she makes a vertical transfer 
from a 2-year to 4-year college or university, proprietary school, or adult 
education program during the reference period. Students transferring 
among institutions at the same level (e.g., among 2-year institutions) 
should not be counted for the purposes of this measure. 

Employment 

A student is considered employed if he or she reports working for pay at 
any point during the reporting period. Employment may include part-time 
or full-time work, and need not be related to a students field of vocational 
study. Individuals should be counted as employed irrespective of whether 
they were continuously working throughout the time period or whether 
they were employed, in the same or different job, prior to high school grad- 
uation. 
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Core Indicator 3: Placement — continued 


Militaiy Enlistment 

Military service may include any branch of the armed forces. 

DATA COLLECTION STRATEGIES 

States are required to report placement outcomes on all vocational program com- 
pleters who graduate from high school. Accordingly, states typically employ one of 
two approaches to data collection: 

Mail or Telephone Survey: State, district, or school staff administer mail or 
telephone follow-up surveys of all vocational program completers at the end 
of the state-designated placement period. 

Administrative Record Matching: State education agencies use a unique 
student identifier, typically the social security number, to electronically 
track secondary graduates as they transition into further education, employ- 
ment, or the military. Data sources include state postsecondary education 
record, Unemployment Wage Record Information, and federal Department 
of Defense records. 

OVERCOMING OBSTACLES TO MAIL AND TELEPHONE SURVEYING 

Low response rates are perhaps the single greatest challenge to conducting follow-up 
information on students. Initial response rates of less than 40 percent are not 
uncommon for the first round of a follow-up survey effort. Consider taking a num- 
ber of steps to increase responses, such as: 
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Core Indicator 3: Placement — continued 


Mail Survey 

• Mailing a postcard two weeks prior to the survey to check for invalid 
addresses and prepare students 

• Providing a postage-paid, addressed response envelope along with the sur- 
vey 

• Offering a sweepstakes prize eligible to all students responding by a given 
date 

• Including a coupon for free or reduced merchandise redeemable at a local 
business 

• Sending a reminder postcard to nonrespondents a week following the dead- 
line 

• Calling nonrespondents 

• Mailing a second survey, along with a letter explaining its importance 

Phone Survey 

• Asking if there is a good time to call back in the event the person is not 
there 

• Requesting forwarding information from the person answering the phone 

• Using state or national databases to track students who may have moved 
within or outside the state 
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Core Indicator 3: Placement — continued 


OVERCOMING OBSTACLES TO ADMINISTRATIVE RECORD MATCHING 

While administrative record matching can reduce some of the effort required in col- 
lecting data, electronic matching systems are not perfect, meaning that placement 
outcomes may be missing for a sizeable percentage of the eligible student popula- 
tion. As such, states performing administrative record matching should plan on 
conducting follow-up mail or telephone surveys to collect information on missing 
students who have bad SSNs, who do not show up in match files, or who have 
information that is clearly inaccurate or invalid. 

To reduce the likelihood of mismatches, states should consider the following steps: 

a. Develop a system of edits. 

To ensure that data are consistently reported, develop a data review system 
that will provide “alerts” to potential problems (e.g., duplicate or incorrect 
SSNs; placements exceeding completers, etc.). 

b. Conduct information audits. 

Conduct periodic audits of record systems of selected agencies with whom 
one has data-sharing arrangements to assess whether electronically reported 
data align with agency records. As an alternative, review internal informa- 
tion audits, or financial audits of state inspector or auditor-general func- 
tions. Regularly compare results over time to note any apparent discrepan- 
cies in magnitude that suggests a problem. 

c. Report performances to locals. 

Share aggregate and individual agency data with locals so that they can 
monitor their own progress as well as understand the value of the informa- 
tion they report. 
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Core Indicator 3: Placement — continued 


d. Establish institutional response thresholds. 

Establish minimum reporting levels that each institution must meet or 
exceed. Provide technical assistance and / or sanctions to institutions that fail 
to meet the minimum reporting rate. 


e. Administer supplemental surveys. 

Develop mail or telephone surveys to supplement electronic matching. 
Survey administration may focus on students who have bad SSNs, who do 
not show up in match files, or who have information that is clearly inaccu- 
rate or invalid. 

f. Develop data-sharing agreements to cover employment in neighboring 
states. 

This can take the form of interstate administrative record matches with 
states that share markets with your state or that attract a large number of 
state students. It could also take the form of working through the Wage 
Record Interchange System (WRIS) or the National Student Loan 
Cooperative. 

g. Link with other administrative record systems. 

Arrange for record sharing with other federal and state agencies, including 
the Department of Defense, United States Postal Service, Railroad 
Retirement Systems, and the Office of Personnel Management. Also, con- 
sider using the Enrollment Verification Service of the National Student 
Clearinghouse. You can learn more about the Clearinghouse by logging 
onto the following website: http://iuioio.nslc.org/ 
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Core Indicator 3: Placement — continued 


CAUTION: 

On January 18, 2001, OVAE released program memorandum 2001-2, entitled 
"The Family Educational Rights and Privacy Act and the Use of State 
Unemployment Insurance Wage Records to Report on Performance." This 
memo, which offered guidance on states' responsibilities under the Family 
Educational and Privacy Rights Act (FERPA), is presently under review by the 
new Administration. Accordingly, states should consult with OVAE staff prior 
to exchanging student record data to ensure the protection of student priva- 
cy. OVAE will provide information to states as soon as it becomes available. 


HOT TIP! 

Consider using the Internet as a way of increasing student survey responses. Survey 
development can often be performed in-house — for example, by offering students 
credit for designing a survey web site. Publicizing the site prior to student gradua- 
tion and offering some form of perk — for example, providing students with an 
alumni page to update their friends on their activities — can serve as an alternative 
to traditional mail and phone survey efforts. For an example, try logging onto the 
following sites: 

Secondary: http:/howw. hemethigh. corn/ forms/ gradsurvey. html 
Postsecondary: http://www. csp. msu. edu/services/followup. him 
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Core Indicator 3: Placement — continued 


FREQUENTLY ASKED QUESTIONS 

Q; Is sampling permitted . ? That is, can agencies collect data on a subset of high school 
graduates completing vocational programs rather than all students ? 

A: No, sampling is not permitted. State and local education agencies must col- 
lect data on all students who complete a vocational program and who grad- 
uate in the state-designated reporting period. 

Q: Can agencies use student intent questionnaires distributed at graduation or base 
reporting using other strategies ( e.g, teacher-reported outcomes) to measure student 
placement ? 

A: State and local agencies may not use statements of student intent to assess 
placement; measurement must reflect actual student outcomes collected at 
the end of the state-designated time period. States may consider using other 
strategies for collecting data, such as teacher-reported outcomes, so long as 
these measurements are defensible and reflect actual student outcomes. 
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Core Indicator 3: Placement — continued 


Tips for Conducting the Survey Process 


Issues for Consideration 


Plan Survey Effort 

/ Identify who will lead survey effort 
/ Determine cost of survey options 
/ Develop survey timetable 
/ Identify objectives of survey 


Can the survey effort be organized and staffed by volunteers? 

What is the size of the graduating population? Cost is directly relat- 
ed to the survey scale. 

What types of information will you need to improve program provi- 
sion or planning? Use the survey to your advantage. 

Don't forget to budget for follow-up of non-respondents 


Create Survey Instrument 

/ Review other surveys for ideas 

/ Develop questions that elicit a range 
of student responses (e.g ratings, 
open-ended, multiple choice) 

/ Pilot test the draft instrument to see 
if the instructions are clear 


What type of survey instrument are other LEAs in your state using? 

What types of information do you need to drive your own program 
improvement efforts? 

Does the survey meet the state guidelines for federal reporting? 

Are the instructions clear and written at a level that students can 
understand? Check by having a group of students complete the sur- 


Is there a need to translate the survey into different languages? 

How long does it take to complete? The shorter the survey the high- 
er the response rate. Try and keep the survey to 15 minutes or less. 


Prepare for Administration 

/ Explain purpose of survey to stu- 
dents before they graduate 

/ Collect contact information on all 
eligible students (phone 
numbers/addresses) 

/ Train phone interviewers 


Do the students understand why they are being surveyed? Explain 
the importance of the effort and how the information will be used. 

Collect contact information, including home telephone numbers and 
addresses, to ensure students or a family member can be contacted. 

Consider collecting contact information on grandparents or other 
family members who are likely to know where the student is resid- 
ing. Alumni associations and school administrative or teaching staff 
are also good sources of information. 

Translate the survey into appropriate languages for your target pop- 
ulation. 

How accurate are responses? Train interviewers in how to administer 
the survey. 


Collect the Data 

/ Distribute surveys or contact stu- 
dents based on the state-defined 
time period 

/ Provide incentives for people to 
return their survey 

/ Follow up among nonrespondents 


When are surveys and/or phone contacts made? Try to time your 
collection as close to the end of the state-defined time period so 
that student recollections are fresh. 

When using surveys, give students an incentive to respond in a 
timely manner. 

Develop phone survey scripts to provide interviewers with consistent 
direction. 

Plan to contact nonrespondents within two weeks of survey due-dates. 

Conduct follow-up contacts for all nonrespondents: 

BE PERSISTENT! 
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Core Indicator 4: Nontraditional 


MEASUREMENT OBJECTIVE 

All secondary students should have the opportunity to pursue studies in a vocation- 
al education program area of their choice, including those that are nontraditional 
for their gender. To ensure all students have access to vocational programs, Congress 
requires state and local education agencies to track student participation in and 
completion of career and technical education programs that lead to nontraditional 
training and employment. 


MEASURE CONSTRUCTION 

Participation and completion rates are based on the number of high school students 
who enroll in or complete a state-identified program associated with nontraditional 
employment in the field. Rates are calculated using the following formula: 

Participation 

Numerator: Number of students in underrepresented gender groups who 
participated in a nontraditional secondary program in the reporting year. 

Denominator: Number of students (male and female) who participated in 
a nontraditional secondary program in the reporting year. 

Completion 

Numerator: Number of students in underrepresented gender groups who 
completed a nontraditional secondary program in the reporting year. 

Denominator: Number of students (male and female) who completed a 
nontraditional secondary program in the reporting year. 
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Core Indicator 4: Nontraditional — continued 


DEFINITIONS 

Nontraditional Training and Employment 

Occupations or fields of work, including careers in the computer science, 
technology, and other emerging high skill occupations, for which individu- 
als from one gender comprise less than 25 percent of the individuals 
employed in each such occupation or field of work. 

Nontraditional Vocational Program 

A vocational program area that addresses occupational areas in which 
underrepresented gender groups comprise less than 25 percent of employed 
persons. 

Crosswalk of Occupations and Vocational Programs 

A list that associates occupations or fields of work that are identified as 
nontraditional in the labor market with the vocational program areas that 
prepare students for entry into these fields. 


DATA COLLECTION STRATEGIES 

States are required to report counts of the number of students participating in, and 
completing nontraditional vocational program areas. To assist LEAs in identifying 
students, states often provide LEAs with a crosswalk that relates occupations that 
are nontraditional for each gender with the vocational program areas, courses, or 
clusters that prepare students for entry into these occupations. 


Core Indicator 4 | Page 2 




Core Indicator 4: Nontraditional — continued 


STRATEGIES FOR IMPROVING DATA QUALITY 

Challenge: What should I do if my state has not identified a set of nontraditional occu- 
pations and/or provided LEAs with a consistent crosswalk of occupations and vocational 
educational programs l 

Solutions: 

• Consult with state vocational administrators to identify national or state 
data sources that can be used to identify occupations that are out-of-bal- 
ance in the workplace, and develop a consistent set of guidelines to assist 
LEA in identifying vocational courses and program areas that are nontradi- 
tional. Use the handout “Strategies for Identifying Nontraditional 
Vocational Programs” at the end of this unit to assist in developing state 
criteria. 

• Develop a list of vocational course or program numbers, coded using a 
standardized state instructional classification system, which LEAs can use to 
identify instructional programs that are nontraditional. 

• If no standardized state instructional classification system exists, consider 
developing descriptions of the skills associated with state-identified, nontra- 
ditional occupations. This will assist LEA staff in associating their instruc- 
tional coursework with nontraditional occupations. Consult the 
Occupational Information Network (O' NET) developed by the U.S. 
Department of Labor for a description of job skills associated with specific 
occupations. The complete list can be found at the following website: 
http://www.doleta.gov/programs/onet/ 
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Core Indicator 4: Nontraditional — continued 


Challenge: It appears that some LEAs have difficulty identifying students participating 
in vocational courseioork or program areas associated with occupations that are out-of- 
balance. How can I improve reporting ? 

Solutions: 

• Provide LEAs with definitions of key state terms, including vocational par- 
ticipant, vocational completer, and nontraditional vocational program area, as 
well as clear descriptions of vocational program areas that prepare students 
for nontraditional occupations or careers. 

• Encourage LEAs to use individual student record data, harvested from stu- 
dent transcripts, to identify individuals participating in nontraditional 
vocational studies. 

• Develop clear, written instructions that LEA staff — including administra- 
tors or teachers of vocational programs — can consult when counting the 
numbers of students participating in or concentrating in nontraditional 
instructional programs. 

Challenge: How can I ensure that each LEA accurately classifies nontraditional students 
over time l 

Solutions: 

• Encourage LEAs to compare state crosswalks against their own course offer- 
ings each year to ensure that state-identified instructional programs corre- 
late with local record systems and account for any changes over time. 

• Ask LEAs to compare their annual counts of student participation and 
completion against prior year data to see if there are any differences that are 
difficult to explain. 

• Suggest that LEAs assign a single individual to collect and enter data each 
year to lower the likelihood of mistakes and to routinize reporting. 
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Core Indicator 4: Nontraditional — continued 


• Hold annual state trainings geared toward new LEA staff who are responsi- 
ble for collecting Perkins data. 

• Develop clear written instructions that stipulate procedures for collecting 
and reporting data. 

Challenge: Staff at many LEAs view reporting as a bureaucratic exercise and do not 
take the time or effort to collect quality data or improve student access to nontraditional 
programs. What can I do to promote buy-ini 

Solutions: 

• Provide LEAs with summary state data that will enable them to evaluate 
their own success in improving student access to nontraditional vocational 
program areas. 

• Share state data with parents, legislators, and the press. 

• Consult with associations, such as the National Alliance for Partnerships in 
Equity, to identify strategies and materials that will support teachers and 
administrators in reforming instructional strategies. 

• Provide technical assistance workshops at the LEA level to communicate 
the importance of equity and to help institution staff understand why they 
are collecting data. 
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Core Indicator 4: Nontraditional — continued 


FREQUENTLY ASKED QUESTIONS 

Q; May I use course enrollments to identify nontraditional areas of study in lieu of state- 
identified ones? 

A: No, Perkins is quite specific in requiring that nontraditional programs be 
identified based on the composition of the workforce. As such, LEAs must 
use state-identified crosswalks or program lists to identify vocational 
instructional areas that are associated with occupations that are out-of-bal- 
ance in the workplace. This may mean that, in some instances, enrollments 
in local programs may not appear out-of-balance. 

Q; What should I do if my state does not provide me with a list of program areas or 
courses that are nontraditional, but only a list of occupations that are out-of balance in 
the workplace? 

A: In the event that your state does not provide clear guidelines for identifying 
nontraditional instructional areas you will need to develop some means of 
relating occupations with vocational programs offered in your agency. The 
preferred approach is to use the Classification of Instructional Programs, a 
guide developed by the U.S. Department of Education that describes the 
vocational coursework that corresponds to a range of occupations in a given 
field. Electronic copies of the report are available on the U.S. Department 
of Education website at the following address: 
http-.Hnces. ed.gov/pubsearch/pubsinfo. asp?pubid=91396 

To assist your state in interpreting your data, you should include a copy of 
your classification scheme with the data you submit and use the same 
methodology over time. Irrespective of the approach you use to identify 
vocational programs, you should be consistent in your measurement over 
time to ensure data are comparable across years. 
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Core Indicator 4: Nontraditional — continued 


Q; What constitutes participation in a nontraditional program areal 

A: A vocational participant describes a student who enrolls in a vocational 
program area or course that prepares individuals for entry into a nontradi- 
tional occupation, as identified by your state. 

Q: What constitutes completion of a nontraditional program areal 

A: A vocational completer describes a student who fulfills a set of state-defined 
criteria that signifies that he or she has mastered a set of academic and / or 
technical skills to prepare him or her for future education and career suc- 
cess. Consult the definitions developed by your state to determine what 
constitutes completion in your state. 

Q; Which students should be included in the denominator for this measurel 

A: The denominator of this measure should include all students, male or 

female, who participate in or complete a vocational program area or course 
designated as nontraditional by your state. 
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Core Indicator 4: Nontraditional — continued 


A GUIDE TO CROSSWALKING NONTRADITIONAL OCCUPATIONS 

AND PROGRAMS 

Step 1 : Identify Occupations That Are Nontraditional in the Workforce 

In collaboration with state vocational education administrators, identify a set of 
occupations — based on state or national data — that are nontraditional for either 
sex. State-specific occupational data can typically be obtained from your state’s 
department of economic development or other employment agency. To assist states, 
OVAE has identified nontraditional occupations based on national data collected by 
the U.S. Department of Labor, Bureau of Labor Statistics, a copy of which is 
included in this handout. 

Step 2: Identify Work Skills Associated With Nontraditional Occupations 

Begin by identifying the skills associated with each nontraditional occupation. Skill 
lists can be obtained by reviewing O ' NET, developed by the U.S. Department of 
Labor. Access the site by entering the following URL on your Internet browser: 
http://ioiow.doleta.gov/programs/onet/ 

Next, search for the occupation you’ve identified as nontraditional and identify the 
skills that are required for workers in this field. Alternatively, you may consult with 
industry associations or educators in your state to identify the skills required for 
success in a given nontraditional occupation. 

Step 3: Crosswalk Nontraditional Occupations with Vocational Education 
Programs 

Identify vocational education programs within your state that prepare students for 
entry into the nontraditional occupations you identified above. Depending upon 
your state, you may have a number of options to use to associate occupations with 
vocational programs in your state. 
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Core Indicator 4: Nontraditional — continued 


I : State Classification Systems 

If your state maintains a standardized classification system for vocational 
education that all LEAs use to code courses, then you may want to base 
your crosswalk on this system. For each nontraditional occupation, link the 
occupational skills you identified with a vocational program area code iden- 
tified by your state. Ideally, each occupation will correspond to a single 
course sequence; however, don’t be surprised if one vocational education 
program area prepares students for multiple occupations. Consult with state 
vocational education curriculum experts if you are not sure of the skills 
taught within a given vocational sequence. 

2: National Classification Systems 

If your state relies on local agencies to develop their own course and pro- 
gram codes, you may want to consider using the Classification of 
Instructional Programs (CIP) codes developed by the US Department of 
Education to crosswalk occupations with vocational education programs. 
You may access the most current CIP by entering the following URL on 
your Internet browser: http://nces.ed.gov/npec/papers/cipPreface.html 

For each occupational skill, search the CIP for the vocational program area 
that provides students with the skills required for success in the nontradi- 
tional occupations you have identified. 

Step 4: Develop and Circulate Instructions to LEAs 

Using the list of vocational programs you identified above, develop written guide- 
lines to assist LEAs in identifying nontraditional programs. Ideally, these instruc- 
tions will contain a list of course codes or descriptions of vocational programs that 
will enable all LEAs in the state to report on students participating in similar cours- 
es, irrespective of the program classification system used locally. 
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Core Indicator 4: Nontraditional — continued 


Strategies for Identifying Nontraditional Vocational Programs 


Has your state identified 
nontraditional occupations? 



Has your state crosswalked 
the identified nontraditional 
occupations to vocational 
programs that prepare 
students for entry into 
these fields? 



Have lists of course or 
program codes been 
shared with the field 
in written form? 



m — ► 


Consult with state staff to identify 
nontraditional occupations. Use 
national or state data to identify 
occupations. 

(See list of nontraditional 
occupations developed 
by OVAE.) 


Does your state have a standardized vocational 
course and/or program classification system? 





Develop a list of vocational 


Identify skills associated 

courses and program areas 


with nontraditional 

that prepare students for 


occupations that each 

entry into nontraditional 


LEA can use to crosswalk 

occupations. 


its own programs. 

(See Guide to Crosswalking.) 


(See Guide to Crosswaiking.) 


Provide on-going 
technical assistance 

(See Program 
Improvement 
Strategies.) 


Develop written 
guidelines and 
circulate to LEAs. 
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Appendix C 



PROGRAM 

$ 

QUALITY 


Improving Performance: 
A Five-Step Process 

A Quick Reference Guide 


U. S. DEPARTMENT OF EDUCATION 

DIVISION OF VOCATIONAL 
AND TECHNICAL EDUCATION 



Step 1: Document Performance Results 


Why document student performance? The first step in improving educational programs and student 
success is to find out how students performed and continue to perform in school and college, and how 
performance varies for different groups of students. In this way, student performance data can help 
educators and stakeholders (a) understand what drives students’ success, (b) assess the quality of edu- 
cation provided by the public system, and (c) identify improvement priorities and strategies. But 
documenting performance is not enough: it is equally important to collect information from different 
sources to not only arrive at a complete and accurate picture of student performance, but to also iden- 
tify gaps in performance and areas for improvement. This section helps you with the following steps: 

□ What to document and how to document state performance results, 

□ Simple and effective tools to display and describe student performance, 

□ How to evaluate the quality of your performance data, and 

□ How to use your performance documentation to establish program improvement priorities. 

What to Document 

Let us assume that you have acquired the performance data you need for the core and sub-indicators. 
To derive the greatest benefit from the data you have collected you need to go further than simply re- 
porting your performance. A good starting point is to group your data in the following different ways: 

• Make comparisons within groups. Comparing subgroups will allow you to further examine 
differences within the group and to hypothesize reasons for, and solutions to, those differences. 

• Make use of benchmarks. Benchmarks provide a point of reference to assess performance rela- 
tive to similar groups. 

• Trends over time. Strive to pinpoint trends by documenting how groups and subgroups are 
performing over time. 


How to Document: Methods and Tools 

There are many ways in which to present performance data. The following suggested methods and 
tools range from fairly simple tables to more sophisticated statistics and graphs: 

• Tables'. Although tables are a nice way to capture a lot of information, they may not always be 
the most effective way to portray the meaning of your data. 

• Graphical displays : Figures accompanied by summary statistics often provide a more expres- 
sive representation of patterns in your performance data and convey a clearer picture of per- 
formance outcomes and gaps. Further, graphics and summary statistics are good supplements 
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to tables. Graphical displays typically include histograms, pie charts, line graphs, and bar 
charts. Summary statistics include the average, median, range, percentile ranking, and standard 
deviation. 


How Do I Know I Can Trust my Data? 

Since all data are limited in some respects, it is important that you consider the quality of your data 
when performing your analyses to ensure that what you say is happening is a true reflection of student 
performance and not simply a problem of limited data quality. According to the criteria established by 
the Office of Vocational and Adult Education to determine data quality, states should do two things 
when looking at data quality: 1) identify major limitations, and 2) assess the implication for interpret- 
ing results. 

Please remember that data quality limitations should not keep you from using the data. Simply recog- 
nize that there may be flaws in the documented data and be cognizant of how you characterize and 
present the data. The data quality improvement process should occur alongside program improvement 
efforts, and each of these processes should influence the other. Also, working with the data will assist 
you in discovering areas on which to focus data quality improvement efforts. And, the better the data 
quality is, the stronger the argument to use data for decision-making. 


Criteria for Establishing Improvement Priorities 

Now that you have documented and analyzed your performance data, a logical next step is to use the 
following criteria to draw up a list of where gaps exist for each core indicator and to identify im- 
provement priorities. 


Criteria 

Tips 

Size of Gaps 

Identify where performance differences are the largest for each core indi- 
cator. 

Trends in 
Performance Gaps 

Over time, are gaps getting larger or smaller? Are changes over time ex- 
plained by unusual events or do gaps reflect an ongoing performance re- 
sult? 

Concentration of 
Gaps 

Identify whether a particular program, population, or school/college dis- 
proportionately impacts state performance for each core indicator, either 
positively or negatively. Are there gaps that cut across measures? Can you 
identify performance pertaining to certain core indicators that is unique to 
particular areas or populations? 

State Improvement 
Priorities 

Determine whether performance areas that you have been targeting for im- 
provement in the past continue to be problematic or have shown improve- 
ment. Also take into consideration what key stakeholders in the state have 
identified as priorities. 
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Step 2: Identify Root Causes 


Your analysis in Step 1 uncovered which students are attaining the desired outcomes and which stu- 
dents are not, but it did not tell you why. Step 2 is designed to address the why questions — why do 
these performance problems and student differences exist? What are the major root causes that deter- 
mine performance and explain student differences in performance? Which of these causes are within 
your control and outside your control? Which of these causes should we address first in our im- 
provement efforts? This section will assist you in identifying and evaluating the root causes of per- 
formance to help guide your search for solutions. 


Why Search for Root Causes? 

Program improvement is, in part, a search for answers to a very basic question: what causes poor per- 
formance? Root causes are those conditions or factors that directly cause or permit a performance gap 
to occur. Rarely are performance problems caused by a single factor; rather, they are caused by a 
combination of root causes and indirect causes, some of which are beyond the immediate control of 
schools and colleges. For example, effective instructional practices are a direct cause of student aca- 
demic achievement because they have a direct impact on academic achievement. In contrast, teacher 
training is an “indirect” cause because it has an effect on student achievement only to the extent that 
the training results in improved instructional practices in the classroom, which, in turn, affect aca- 
demic achievement. 


Indirect Cause Direct (Root) Cause Performance Indicator 



How to Identify Root Causes 

The search for root causes is a search for the most direct and highest impact causes of performance 
gaps on core indicators that are within the control of schools and colleges. Your search should draw 
on current research and evaluation, and use multiple methods and data sources to test specific hy- 
potheses. There are many different approaches to identifying root causes, but most approaches in- 
volve three basic phases: (1) identify potential causes, (2) analyze and evaluate potential causes, and 
(3) select a few critical root causes to address. 
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Step 3: Select Best Solutions 


Once you have identified the most critical root causes to address in your improvement effort, the next 
step is to identify and select the solutions that seem most promising for testing and evaluation. This 
section will assist you in reviewing and selecting potential solutions for testing in Step 4. 


Why Take the Time to Search for and Evaluate Alternative Solutions? 

How do you identify the best solutions? First, identify or develop a full range of potential solutions. 
Selecting a full range of choices stretches your thinking and helps develop more creative solutions. 
Next, select the most promising of these potential solutions. Any systematic analysis of alternative so- 
lutions has two parts: 

1. Reviewing the underlying logic of the solution — is it based on a sound theory of root causes 
(Step 2) and how does the solution address these causes, and 

2. Reviewing the empirical evidence — has the solution worked under similar or comparable 
circumstances to yours, and is the evidence strong and compelling? 


How to Develop Solutions: Improvement Strategies and Models 

In developing solutions, begin by identifying the potential improvement strategies that can impact the 
root and indirect causes in your cause-effect model from Step 2. Next, identify specific school or col- 
lege model practices that are based on these strategies. This can be illustrated by the example below 
for the Secondary Academic Attainment indicator. 


Root Cause 

Improvement Strategy 

Model (Model Practices) 

Time on Task 

School Class Scheduling 

Block Scheduling Model 1 
Block Scheduling Model 2 


Identifying Potential Strategies and Models: Three Methods 

For best results, use multiple methods to identify potential improvement strategies and models. Below 
are three methods that you should utilize: 

1 . Review what others propose 

2. Benchmark peers and leading performers 

3. Develop your own solutions 


Quick Reference Guide 5 






Narrowing the Choices: Assessing and Comparing Alternative 
Strategies and Models 

Since not all solutions you have identified will work for you, narrow the choices by assessing and 
analyzing their rationale or underlying logic and the evidence that supports them. Use the following 
criteria in your assessment. 

• Sound Theory > and Logic: The best solutions are ones based on clearly understood and sound 
theory that explains how the improvement strategy and model works and why. 

• Strong Evidence: The best solutions also are supported by strong evidence indicating they have 
worked under conditions similar to yours, especially in schools and colleges that have similar 
“causes outside your control” identified in Step 2. 

Improvement strategies and models with sound theory and compelling evidence rarely come ready 
made for your situation. Y ou generally have to choose between imperfect alternatives that have some 
limitations in theory and evidence. Therefore, it is necessary to fully compare and contrast the trade- 
offs among alternative improvement strategies and models before choosing which solution or combi- 
nation of solutions to test and evaluate yourself. 
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Step 4: Pilot Test and Evaluate Solutions 


Now that you have identified a set of promising solutions — improvement strategies and models — on 
which to base initial improvement efforts, the next step is to create an evaluation approach that will 
allow you to assess how well the improvement strategies and models are working. This section will 
assist you in selecting practical evaluation designs and analysis tools that you can use to gauge the 
success of your improvement efforts. 


Why Evaluate? 

Even though findings from the literature or your own common sense may suggest you have found a 
winning formula, it is important to pilot test improvement strategies and models to see if they produce 
desired outcomes. Additionally, testing and evaluating improvement strategies provides an evidence- 
based method for justifying your educational improvement efforts and goes beyond theorizing about 
what works — an evaluation will supply you proof of whether the strategy works. 


How to Test Solutions: Designing an Evaluation Strategy 

Conducting an evaluation can seem a daunting task, particularly given that you will simultaneously be 
working to introduce and run a program improvement effort. To lessen the burden, identify in ad- 
vance a research methodology that you can use to collect the necessary information. The following 
are some steps to follow in designing and conducting an evaluation: 

□ Choose a study design. Listed below are three study designs you might consider when struc- 
turing your evaluation. 

1. Option A: Random Assignment with Control Groups 

2. Option B: Comparisons with Similar Populations 

3. Option C: Comparing Individuals Against Themselves 

□ Select pilot sites. While it is tempting to implement your program improvement strategy 
statewide, it is advisable to select a subset of sites in which to pilot-test ideas. 

□ Select outcome measures. To help track performance changes, develop both short- and long- 
term measures to provide some indication of the success of your improvement efforts. 

□ Identify data sources. After selecting short and longer-term outcome measures, it is important 
to identify data sources and collection instruments that will allow assessment of whether the 
improvement strategy is affecting student or program outcomes. The following are some ex- 
amples of data sources: student transcript records, interviews or focus group discussions, state 
standardized test scores, classroom visits or observations, and state or locally administered sur- 
veys. 
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□ Train pilot site staff. Since the outcomes of the improvement effort will hinge on the work of 
participating administrators and school staff, communicate the purposes and activities related 
to the improvement effort to ensure that pilot sites are faithful to the planned intervention. 


Analyzing Initial Results 

Whenever possible, employ basic descriptive or summary statistics — such as the ones described in 
Step 1 — to assess outcomes. Keep in mind the importance of looking at the data from different angles 
to find out whether gains are realized equally across all groups. If outcomes differ, consider whether 
variations are due to the manner in which your solutions have been implemented or due to character- 
istics of the subgroup itself. As you near the end of your pilot project timeline, determine whether you 
are ready to move to full implementation of your program improvement model. 
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Step 5: Implement Solutions 


If you have come this far, you have likely identified a set of solutions — improvement strategies and 
models — that increase student performance in pilot sites. As you prepare to expand the pool of par- 
ticipants, it is a good idea to come up with a monitoring process that will allow you to obtain ongoing 
feedback on the improvement strategy. This section will help you to develop implementation plans to 
monitor outcomes across the full range of implementation sites. 

Moving from Pilot Testing to Statewide Implementation 

Full implementation will require flexibility and a willingness to revise strategies as new sites join the 
effort. Y ou should expect to modify solutions and evaluation approaches to accommodate unique site 
characteristics or unexpected situations that arise in mid-stream. Just as with the pilot, monitor site 
performance on a number of dimensions to ensure you’re achieving the intended results. Plan to 
evaluate both the desired outcomes as well as the process you are using to roll out the implementation 
across agencies. 

Monitoring Ongoing Implementation 

Expanding your improvement efforts requires the development of a comprehensive strategy that will 
enable you to monitor how well the implementation is progressing. Plan to enlist the support of pilot 
site staff to help explain the importance of monitoring to new participants and to serve as field-based 
mentors, for example by having educators from new sites visit pilot sites to speak directly with ex- 
perienced staff. Steps to consider taking for monitoring on-going implementation may include: 

□ Preparing a “Process Evaluation Plan” to help assess the manner in which you are implement- 
ing new improvement strategies, 

□ Establishing short-term outcome measures (e.g., changes in attendance or student grades) that 
can be used to assess on-going efforts, 

□ Identifying process indicators that capture how well improvement activities are being imple- 
mented, 

□ Calling or visiting school and district staff to observe whether they are correctly applying pro- 
posed solutions, 

□ Requiring local faculty and staff to attend technical assistance workshops to discuss their 
observations and challenges, and 

□ Conducting focus groups with students and faculty to assess changes in beliefs or practices. 
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Sustaining Improvement Efforts 


Continuous improvement is a never-ending process that requires that you constantly review and cri- 
tique the outcomes of improvement efforts. To sustain your effort, schedule time — much as you did in 
Step 4 — to revisit your strategy. 
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To: Lewis and Clark -Community College Career Program Students 

From: Jane Fleming, Perkins Special Populations Project Manager 

Subject: Carl D. Perkins III Grant: Special Populations 

As a student enrolled in a career program at Lewis and Clark Community College, you may 
be eligible for services and assistance under the Carl D. Perkins 1 1 1 Grant. Grant funds are 
used, in part, to assist career students at risk of not succeeding in their educational 
pursuits. The federal govenunent has identified certain traits that have prevented 
students from being successful in meeting their educational goals. Through no fault of 
their own, students are often unable to complete their course of study and must drop out 
of school, are unable to attend school, or are not able to do as well in their programs as 
they might be capable of doing because of these traits. The traits that prevent students 
from succeeding are: economic disadvantage, academic disadvantage, 
disability/disabilities, single parent, displaced homemaker, nontraditional, and limited 
English proficiency (LEP). (Please see the attached page for the definitions of each trait.) 
Students with one or more of these traits are referred to as Perkins Special 
Populations Students. 

While Perkins Special Populations Students are eligible for services available to all students 
at LCCC, Perkins can provide additional services to eligible career students. For students 
with needs beyond what Perkins can provide, Perkins has been able to assist those 
students in finding and working with other on- and off-campus agencies in providing help. 

The Illinois State Board off Education and various agencies of the United States require 
educational institutions to maintain information on students pertaining to factors such as 
disability, sex, marital/ parental status, etc. and the type of aid for which a student applies. 
The first step in providing assistance and services is identifying students as Perkins Special 
Populations Students. Please take a few minutes to complete the attached survey. 

Should you choose to complete the survey, please return 9AIA the survey to your 
instructor. Please keep this letter, the Special Populations category definitions, and 
program list for your records. All surveys and information will be kept in strictest 
confidence. Students are not reauired to complete the survey; participation is 
strictly voluntary. 

If you would like to know if you are eligible for assistance/services or if you are in need of 
assistance/services at anytime during the semester, please do not hesitate to contact me. 
To respect your privacy, I shall not contact you except in response to your inquiries. I can 
be reached at my office in Caldwell 4326. My phone number is: (618)468-4020; if I am 
not available, please leave a message on my voice mail and I shall return your call as 
soon as possible. My e-mail address is: jfleming@lc.ce.il. us I look forward to working with 
you during your time at LCCC. 


Source: Jane Fleming, Special Populations Coordinator, Lewis & Clark Community College. 



Perkins Special Populations Survey 


The Illinois State Board of Education and various agencies of the United States 
Government require educational institutions to maintain information on students 
pertaining to factors such as disability, sex, marital/ parental status, etc. and the type of 
aid for which an individual applies. The information requested on this sheet is for 
compliance with certain record keeping requirements and voluntary disclosure is the 
student's choice. The College believes all persons are entitled to equal educational 
opportunities and does not discriminate against its students in their education because of 
race, color, sex, religion, national origin, disability, veteran status, age, marital status or 
any other protected group. 

Lewis and Clark Community College is committed to the most fundamental principles of 
human dignity, equality of opportunity, and academic freedom. This commitment requires 
that decisions involving students and employees be based on individual merit and be free 
from discrimination or harassment in all its forms. Programs services, and employment 
opportunities are administered by Lewis and Clark Community College without regard to 
sex, race, ethnicity, color, creed or religion, national origin, disability, age, marital status, 
military status, sexual orientation, and other protected categories. The College abides by 
affirmative action principles, makes reasonable efforts to accommodate qualified 
individuals with special needs, and complies with all federal and state nondiscrimination, 
equal opportunity and affirmative action laws, orders, and regulations. These include, but 
are not limited to: (a) Title VI of the Civil Rights Act of 1964- (b) Title IX of the Education 
Amendments of 1972; (c) Section 504 of the Rehabilitation Act of 1972 and the Americans 
with Disabilities Act of 1990; (d) the Age Discrimination Act of 1975; and (e) the Illinois 
Human Rights Act. 

Complaints of discrimination prohibited by College policy are to be resolved within the 
existing College procedures. For additional information or assistance on the equal 
opportunity, affirmative action and harassment policies and procedures of Lewis and Clark 
Community College, please contact: George C. Terry, Dean of Student Life and Director of 
Affirmative Action Programs (Title IX, ADA, and 504 Coordinator), River Bend Arena, 

Room 112, Lewis and Clark Community College, Godfrey, IL 62035 618-466-341 1. 

It is the policy of the college that sexual harassment of employees or students on campus 
is unacceptable shall not be tolerated. Any employee or student of the college who feels 
that he/she has been a victim of a sexual harassment should notify the college's Personnel 
Office and the complaint will be investigated. 


Perkins Special Populations Survey 


As a student enrolled in a career program at Lewis and Clark Community College, you may 
be eligible for services and assistance under the Carl D. Perkins 1 1 1 Grant. I n order to 
determine eligibility for Perkins Grant assistance, please complete the following, printing 
your responses clearly. All answers and surveys will be kept in strictest confidence. 

Participation in this survey is strictly voluntaky. 

NAME: 

SOCI AL SECURI TYNUMBER: 

PHONE: 

ADDRESS: 

GENDER: Male Female 

PROGRAM MAJOR NUMBER (see attached list for major numbers): 

If declaring any of the following Allied Health programs (5202A, 5202j, 5205A, 5208A, 
5208J , 5230A), have you been formally accepted into that program? Yes No 

Do you receive any type of financial aid based on need? Yes No 

If yes, please indicate the type, (check all that apply) 

□ Federal Pell Grant 

□ Public Aid 

□ WIA Opportunities 

□ Department of Human Services 

□ Office of Rehabilitative Services 

□ FSEOG 

□ I SAC MAP 

□ I SACHA 

□ Other 

Are you a single parent with dependent children/ child? Yes No 

Dependent child must reside with you. 

Women who are now pregnant and single should answer "yes". 

Are you a displaced homemaker? Yes No 

(see definitions on previous page) 

Do you have any type of physical, learning, or mental disability? Yes No 

If you have a disability, please indicate the type below, (check all that apply) 



□ Learning Disability 

□ Hearing I impaired 

□ Developmental^ Disabled 

□ Emotionally Disabled 

□ Mobility Impairment 

□ Paraplegic 

□ Neurological Impairment 

□ Cognitive Impairment 

□ Vsually Impaired (beyond simply wearing corrective lenses) 

□ Other (please specify) 

I s English your native/ primary language? Yes No 

Did you grow up speaking primarily English? 

I n high school, were you enrolled in a tech- prep program? Yes No 

If yes, what was the tech-prep program? (i.e. business, computers, office procedures, 
agriculture, etc.) 



Perkins Special Populations Survey 


The Illinois State Board of Education and the United States Department of Education 
require educational institutions to maintain information on students pertaining to factors 
such as disability, sex, marital/parental status, etc. and the type of aid for which an 
individual applies. The information requested on this sheet is for compliance with certain 
record keeping requirements. 

Lewis and Clark Community College is committed to the most fundamental principles of 
human dignity, equality of opportunity, and academic freedom. This commitment requires 
that decisions involving students and employees be based on individual merit and be free 
from discrimination or harassment in all its forms. Programs, services, and employment 
opportunities are administered by Lewis and Clark Community College without regard to 
sex, race, ethnicity, color, creed or religion, national origin, disability, age, marital status, 
military status, sexual orientation, and other protected categories. The College abides by 
affirmative action principles, makes reasonable efforts to accommodate qualified 
individuals with special needs, and complies with all federal and state nondiscrimination, 
equal opportunity and affirmative action laws, orders, and regulations. These include, but 
are not limited to: (a) Title VI of the Civil Rights Act of 1964; (b) Title IX of the Education 
Amendments of 1972; (c) Section 504 of the Rehabilitation Act of 1972 and the Americans 
with Disabilities Act of 1990; (d) the Age Discrimination Act of 1975; and (e) the Illinois 
Human Rights Act. 

Complaints of discrimination prohibited by College policy are to be resolved within the 
existing College procedures. For additional information or assistance on the equal 
opportunity, affirmative action and harassment policies and procedures of Lewis and Clark 
Community College, please contact: George C. Terry, Dean of Student Life and 
Coordinator of Affirmative Action Programs (Title IX, ADA, and 504), River Bend Arena, 
Room 112, Lewis and Clark Community College, Godfrey, IL 62035 618-466-341 1. 



Who is Perkins Special Population Student? 


A Perkins Special Populations Student is a student enrolled in a career-technical 
education program and possesses one or more of the following traits: 

1. ECONOMICALLY DISADVANTAGED: An economically disadvantaged student is one 
who receives financial aid based on need. Financial aid based on need includes: Federal 
Pell Grant, Public Aid, WIA, and Opportunities. 

2. ACADEMICALLY DISADVANTAGED: An academically disadvantaged student is at 
risk academically. This characteristic can be identified by low placement scores, academic 
suspension or probation, instructor referral, and/or has received academic assistance in 
the Math Lab, Communications Center, Biology Tutoring, OTEC/Com'puter Sciences Lab 
Assistance, Nursing Lab, and/or Dental Assisting Lab Assistance. Placement test scores 
can be taken from two academic years ('99- '00, '00- '01). 

3. DISABLED: A disabled student is one who is physically, mentally, emotionally, 
learning, andd/or developmental^ disabled/impaired. A disabled student may be 
identified by the student himself on the Perkins Special Populations Survey, the 
Department of Human Resources-Office of Rehabilitative Services, and/or campus service 
providers. 

4. NONTRADITIONAL: A nontraditional student is enrolled in a program in which 
25% or less of that program's enrollment is of the same gender as the student. 

5. LIMITED ENGLISH PROFICIENCY (LEP): A LEP student is one who was not born in 
the United States, comes from an environment where a language other than English is 
dominant, is American I ndian or Alaskan Native and comes from an environment where a 
language other than English has had a significant impact on the level of English language 
proficiency; and by reason thereof, has sufficient difficulty speaking, reading, writing, or 
understanding the English language as to deny such individual the opportunity to learn 
successfully in a classroom where the language of instruction is English or to participate 
fully in our society. 

6. SINGLE PARENT: This student is a single parent with dependent children who 
reside with that student. Single pregnant women are included in this category. 

7. DISPLACED HOMEMAKER: This is a student who: 

1. A. has worked primarily without remuneration to care for a home and 
family, for that reason has diminished marketable skills; 

B. has been dependent on the income of another family member but is no 
longer supported by that income; or 

C. is a parent whose youngest dependent will become ineligible to receive 
assistance under Part A of Title IV of the Social Security Act (42 U.S.C. 601 
et seq.) not later than 2 years after the date on which the parent applies 
for assistance under this title; and 



is unemployed or underemployed and is experiencing difficulty in obtaining 
or upgrading employment. 
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November 20, 2001 


Bill Anderson 

U.S. Office of Personnel Management 
Office of Workforce Information Room 7439 
1900 E. Street NW 
Washington, DC 20415 


Dear Mr. Anderson: 

This letter is a request for your assistance. In order to satisfy the requirements of 
the Carl D. Perkins III federal legislation, the Illinois State Board of Education 
(ISBE) needs to report post-program outcomes to the U.S. Dept, of Education 
regarding former students. Many of these students obtain Federal employment 
following their graduation from Illinois schools. The ISBE requests to arrange a 
data match between the student records from Illinois and employment records 
from the U.S. Office of Personnel Management (OPM) to identify former students 
now in Federal employment. Enclosed is a Memorandum of Understanding for 
your review. Please inform us as to your requirements for this match to occur. 

If the requested data match is approved, the preferred method of file transfer is 
over the internet using a password-protected ISBE FTP site. Alternatively, 
computer diskettes can be sent via postal mail or Federal Express. At this time, 
there are two student record files for which matching is sought. One contains 
approximately 84,000 secondary records, and the other approximately 43,000 
post secondary records. The format of the files is fixed length ASCII. 

• Columns 1-9 are SSN 

• Columns 10-14 are School ID 

• Columns 15-24 are filler 

• Column 25 is cohort level (“P” or “S”) 

• Columns 26-27 are Cohort year (00) 


The data fields that the ISBE seeks in return are the following: 



Field 

Columns 

orig file 

1-27 

ssn 

28-37 

agency code 

38-41 

agency name 

42-93 

pay plan 

94-95 

grade level 

96-97 

occ'l code 

98-101 

code 


translation 

102-153 

basic pay 

154-159 

work sched 


code 

160 

State code 

161-162 

State name 

163-180 

date of hire 

181-186 

date of exit 

187-192 


Following the match, the files on would be returned using one of the above 
mentioned methods with an explanation of the new file layout. Thank you for 
your help and attention in this matter. If there are any questions or matters to 
resolve, feel free to contact Barry Pedersen at 217/782-4620. 

Sincerely, 


Fran Beauman, Division Administrator 
Workforce Preparation Partnerships 



Memorandum Of Agreement Between The 
Illinois State Board Of Education And The 
Office of Personnel Management 


This agreement, made the day of 2001, between the Illinois State 

Board of Education (ISBE) and the Office of Personnel Management (OPM). The 
purpose of this agreement is to designate the OPM as an “authorized agent” of the ISBE 
for purposes of obtaining and reporting information concerning the placement and 
retention of students in employment as required by section 1 13 of the Carl D. Perkins 
Vocational and Technical Education Act of 1998 (Perkins III). 

RECITALS: 

1 . The ISBE maintains education records on enrollees that include information on 
student demographics, programs of study, achievement, attainment and social security 
numbers. Such records also include information on students participating in 
vocational education programs. The OPM maintains employment wage records for 
federal workers. 

2. Perkins III (P. L. 105-332) creates a State performance accountability system for 
vocational education programs. States must report annually to the U.S. Department 
of Education (USDE) on the progress of the state in reaching agreed upon levels of 
perfonnance on core indicators specified in the law. These core indicators include 
placement and retention in employment. 

3. In order to determine employment outcomes for those vocational education 
students included in Illinois’ Perkins III accountability system, the ISBE will supply 
the OPM with a list of the social security numbers of these students. The OPM will 
access its wage records using these social security numbers and determine 
employment outcomes for any of these students present in its database. The OPM 
will report the results of this analysis to the ISBE. 

4. The Family Educational Rights and Privacy Act (FERPA) generally prohibits the 
disclosure of education records without the consent of the parent for children under 
the age of 18 or from students attending postsecondary educational institutions. 

Under FERPA, education records are defined as records directly related to a student 
and maintained by an educational agency or institution. The records accessed by the 
OPM to meet Perkins III performance reporting requirements are education records, 
and subject to FERPA. 

5. FERPA contains several exceptions to the general rule that education records may 
not be disclosed without prior, written parental consent. One exception allows for 
disclosures to authorized representatives of the Secretary of Education, the 
Comptroller General, the Attorney General, and state and local educational 
authorities. Such a disclosure must be made in connection with an audit or evaluation 
of a Federal of State supported education program. The disclosure may also be made 
for the enforcement of or compliance with Federal legal requirements related to the 
Federal or State education program. 



6. The disclosure of personally identifiable student information by local education 
agencies and educational institutions to the OPM is for the purpose of complying with 
the performance reporting requirements of Perkins III, and is permissible under 
FERPA. The USDE has concluded that the OPM can be designated an authorized 
representative for purposes of compiling and reporting infonnation as required by 
Perkins III. 

7. Without access to these records, the state of Illinois will not have the most 
complete and accurate perfonnance information possible for compliance with Perkins 
III reporting requirements. 

AGREEMENT 

1 . The ISBE designates the OPM as its “authorized representative” under FERPA 
for the limited purpose of analyzing student information provided by ISBE in order to 
comply with the performance reporting requirements of Perkins III. This 
authorization is limited to data obtained from the education records (as defined by 
FERPA) of secondary and postsecondary education students in Illinois. It is 
understood and acknowledged by the parties that the OPM will not redisclose any 
personally identifiable infonnation to other than the ISBE. 

2. The ISBE will deliver to the OPM electronic data files containing student records 
for analysis to detennine their possible post-program employment status in the OPM 
database. 

3. The OPM agrees to destroy all personally identifiable infonnation, such as social 
security numbers, obtained from the above-referenced education records as soon as 
the results of the analyses performed have been delivered to the ISBE, or when the 
information is no longer needed, whichever comes first. All versions of such 
information and data, electronic, paper, or otherwise, must be destroyed. 

This agreement shall be in effect for years from the date of the last signature. 


ISBE 


Date 


OPM 


Date 
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Illinois State Board of Education 
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Ronald J. Oldwltc Olann W. McOto 

Chairman S!m Suparmtonoentot tauMtton 

Dooember 17, 1999 


Ms. Diana Robinson 
Associate Superintendent 
Business. Community and 
Family Partnerships 
Illinois Slate Board of Education 
100 North First Street 
Springfield, Illinois 62777-0001 

RE; Tech Prep Student Data 

Door Ms. Robinson: 

Uur office is in receipt of your letter of December 1 , 1 999, wherein you inquire about 
the transfer of student information from secondary schools to community colleges. 
Specifically you ask whether high schools with Tech Prep programs can transfer student* 
specific educational records to post-sseundary institutions for educational purposes? 

• The brief answer to your questions is yes, but only with the written consent of 
the parent or eligible student and consistent with the parameters outlined below. 

The release of student information is governed by Stale and federal statutes, namely, 
the Illinois School Student Records Act (105 ILCS 10/6) and the Family Educational 
Rights and Privacy Act (20 U.S.C.A. 1232g). The protection of individuals' right# to 
privacy cautions restraint and reserve in the disclosure of student records. On the other 
hand, Ihe concern for enhanced uocesa to student records by students, parents and other 
parties, heralds a shoring of information. In an effort to harmonize these two policy 
interests Illinois and federal law allow for access to school student records pursuant to an 
express release by the parent and/or parental designee [105 IT.C.S 10/5 (a)], yet generally 
prohibit the transfer, release, disclosure, or dissemination of school student records 
except under limited circumstancoe. Outlined below is the release language contained in 
Section 5 und the relevant exceptions embodied In Section 6 of the Illinois School 
Student Records Act: 

Sec. 5 (a) A parent or any person specifically designated as a representative by a 
parent shall have the right to inspect and copy all school studont permanent and 
temporary records of that parent’s child. 

Sec.6. (a) No school student records or information contained therein may be 
released, transferred, disclosed or otherwise disseminated, except as follows: 
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(2) To an employee or official of tilts school or school district or State board with 
current demonstrable educational ot administrative interest in the student, in 
furtherance of such interest; 

(3) To the official records custodian of another school within Illinois Or on official 
with similar responsibilities of a school outside Illinois, in which the studeut hat 
enrolled, or intends to enroll, upon the request of such official or student; 

(8) To any person, with the prior .specific dated written consent of the parent 
designating the person to whom the records may be released, provided that at the time 
any such consent is requested or obtained, the parent shall be advised in writing that 
he lias the right to inspect and copy such records in accordance with Section 5, to 
challenge their contents in accordance with Section 7 and to limit any auoh oonsent to 
designated reootds or designated portions of the information contained therein. 

Note that Section 10/6 donee with two caveats. 

(b) No information may be released pursuant to [subparagraph (3) above] ... 
unless the parent receives prior written notice of the nature aod substance uf the 
information proposed to he released, and an opportunity to inspect and copy such 
records in accordance with Section S and to challenge thoir contents in accordance 
with Section 7. 

(d) Exoept for the student and his parents, no person to whom information is 
released pursuant to this Section and no person designated as a representative by a 
parent may permit any other person to have access to such Information without a 
prior consent of the parent obtained in accordance with the requirements of 
subparagraph (8) of paragraph (a) of this Section. 

The Federal Family Educational Rights and Privacy Act (FERTA) (20 U.S.C.A. 
1232g) contains parallel provisions with regard to the release of school student records. 
FFRPA provides for a general parental oonsem/release accompanied by exceptions for 
school officials attire .scliool(s) of current and prospective enrollment. 

The 12 fl ' grade student specific data yon request to be transferred is statutorily defined 
as “school student record” or “any writing or other recorded information concerning a 
student and by which a student my be individually Identified../' [105 ILCS 10/2(s).] As 
such, it in ococarible in cither of two ways: 1) via an express release or consent from 
parent, or as appropriate, student or 2) pursuant to a relevant exception, if applicable. 
Implementation of option one entails having parents) and/or Tooh Prep student, os 
appropriate, sign a form authorizing the release or transfer of the student data to 
community colleges which in turn, may create an early identification system to assist 
Tech Prep students with their transition to post-secondary study. 

Note diut die underpinnings of Student Records law provide that rights of access arc 
Initially enjoyed by the parent of the student and terminate when a student is emancipated 
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at eighteen years of uge or graduates from secondary school, embarks upon marriage or 
military service, whichever occurs first. Thereafter, the statutory rights arc exclusively 
the scudem’s [105 ILCS 1U/Z (g)J, Thus, parental consent will be required to release the 
student records of those 16 tn-17 year old Tech Prep * indents who have not yet finished 
their high school oourscworlc. Parental release is not rcquirod, however, for emancipated 
students who have boon aecordod ull statutory righto and privileges. 

Regarding option two, a perusal of potentially relevant exceptions reveals that 
exception two cited above is in fact inapposite because the information is not being 
transferred to an official or employee of the school or school district where the Tech Prop 
student is currently enrolled. Instead, the transfer to be effectuated Is to another school; 
under such circumstances exception three has potential applicability. 

Upon scrutiny, however, it is apparent that the facta of the instant case do uot 
conform to this exception either, became as we understand, Tech Prep students have not 
yet enrolled nor expressed intent to matriculate at a specific post-secondary institution. 
In light of tho foregoing, it appear that the most viable way for l ech Prep high schools 
to transfer student-specific data is by obtaining a release from the parent, or student, as 
applicable. 

Thank ym> for sharing your concerns and questions with the lllimm Slate Board of 
Education. Plcane contact me or Ms. Gewghmu fheoharis, Assistant Legal Advisor, at 
(312) 814-223G. 



Michael J, Hems 
General Counsel 
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FERPA (Family Educational Privacy Rights Act) 

Questions; Can students identified as Tech Prep in high school be identified to 
community colleges before those students enroll at the college? If they can, how 
much information on that student can be given to the college? 

Answer: Maybe. If the student is planning or seeking to enroll at the institution, the 
Tech Prep student can be identified to authorized representatives or slate educational 
authorities in connection with the audit and evaluation of Federally-supported education 
programs, or in connection with the enforcement of the Federal legal requirements which 
relate to such programs. 

If the Tech Prep identification is part of the high school directory information, such as 
name, address, programs, interests, awards, extracurricular activities, the high school can 
give the information to the community college without consent. The high school would 
determine what is directory information. 

If Tech Prep identification is not part of the directory information, written consent may 
be given by the parent for the student under 18 years old or by the student himself/herself 
who is 18 years or older. The consent must list the specific information that can be 
released as part of directory information. The designated directory information must be 
posted or advertised so those individuals not wishing to release the information can have 
the option. 

The other option is for high schools to give information of all students to authorized 
representatives or stale educational authorities at the community college as long as the 
information collected shall be protected in a manner which will not permit the personal 
identification of student and their parents. The community college would have to identify 
Tech Prep students from the list. 

Source: Personal Communication, Kathy Wolen from Family Policy Compliance Office, 
U. S. Department of Education. Telephone (202) 260-3887 
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Chapter Six 

Tech-Prep Education 

Tech-Prep programs in California will be accountable, integrated and articulated 
curricular pathways among secondary and postsecondary programs that result in 
students earning advanced technical degrees and certification. These pathways will 
be academically rigorous, and provide students with not only the fundamental skills 
that are required for postsecondary admission, but also the technical skills that will 
enable them to have successful careers in California's new, highly technical 
economy. 

California's future Tech-Prep delivery system will build upon the infrastructure and 
programs that have already been developed in the state (i.e. elements of the School 
to Career system; the current Tech-Prep Local Consortia and centers; etc.). To 
ensure that Tech-Prep is successful in creating accountable Tech-Prep programs that 
are linked to careers in the new economy, the Tech-Prep delivery system will have 
two major components: 1) State Administration and Leadership, and 2) Tech-Prep 
Local Consortia. The organization around these components will ensure: 

• Linkage of programs with the local, state and regional economies. 

• Collaboration and systematic articulation of programs among high schools and 
community colleges. 

• Development of comprehensive strategies among multiple state and federal 
programs to encourage joint planning and avoid unnecessary duplication of 
service delivery. 

• Lunding and programmatic decisions directed toward industry priorities. 

• Locus on accountability, and an infrastructure for monitoring effectiveness. 

• Capacity for researching and identifying effective programs and practices linked 
to academic and industry standards. 

• Locus on new and emerging technical occupational areas. 

State Administration and Leadership 

The California State Department of Education and the Chancellor's Office of the 
California Community Colleges will jointly administer Tech-Prep. The leadership 
and staff of the two agencies will convene a Joint Management Team to draft a 
Memorandum Of Understanding (MOU) including the principles, decision-making 
processes, and operational parameters to guide its operations. The MOU will be 
broad in scope, including the relationships and operational parameters between the 
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two agencies regarding the overall operations related to the Carl D. Perkins 
Vocational and Technical Education Act (VTEA). 

The Chancellor's Office, California Community Colleges, and the California 
Department of Education will collaboratively have administrative responsibility for 
the distribution of funding to the Tech-Prep Local Consortia. Technical assistance 
and monitoring will be conducted collaboratively by the two agencies, and 
coordinated through the Joint Management Team. The Joint Management Team will 
be responsible to: 

• Devise a transition plan to re-align the Tech-Prep program with the particulars of 
the approved VTEA State Plan. 

• Establish state performance measures and indicators, in collaboration with field 
representatives, and approved by the Joint Advisory Committee (JAC) of the 
governing boards. 

• Review and certify funding awards for Tech-Prep Local Consortia and distribute 
funds to the administrative fiscal agents. 

• Develop articulation guidelines that ensure systematic coordination with 
segmental and statewide standards. 

• Provide policy guidance to the Tech-Prep Local Consortia in developing strategic 
plans, accountability, funding, and other administrative and programmatic 
issues as necessary. 

• Regularly meet and confer with representatives of Tech-Prep Local Consortia. 

• Review and summarize annual accountability reports submitted by the Tech- 
Prep Local Consortia. 

• Periodically monitor Tech-Prep Local Consortia for fiscal and programmatic 
compliance. 

• Recommend annually to the JAC whether or not to continue funding of Tech- 
Prep Local Consortia, based on fiscal and performance reports and monitoring 
findings. Where the JAC determines that performance is deficient, it may elect to 
require improvement plans and monitoring and/or technical assistance. 

• Produce required state and federal reports and conduct other functions as 
determined necessary by the state agencies. 

• Tech-Prep Local Consortia shall be composed of secondary and postsecondary 
institutions, industry and labor representatives, and other key stakeholders 
within the region. 
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• Assist the state agencies in their periodic monitoring of Tech-Prep Local 
Consortia for fiscal and programmatic compliance. 

• Collaborate with the state agencies on determining performance indicators, 
measures, and levels of performance. 

• Provide technical assistance to Tech-Prep Programs to ensure that Tech-Prep 
Program course sequences integrate academic and vocational and technical 
curricula, programs are articulated among the secondary and postsecondary 
segments, and students are completing Tech-Prep Programs. 

• Coordinate Tech-Prep with other Carl D. Perkins funded programs and other 
Education and Workforce preparation efforts, such as Partnership Academies 
and other allied initiatives. 

Tech-Prep Local Consortia 

Tech-Prep Local Consortia need to be the heart and soul of the vision for Tech-Prep 
in California - the locus of the teaching and learning that all the other resources are 
directed to support. Currently, 80 Tech-Prep Local Consortia operate in California, 
and they provide an infrastructure upon which to improve and expand Tech-Prep 
Programs. To achieve accountable outcomes, Tech-Prep Local Consortia must 
develop programs that are more systematically articulated. Program - to - program 
articulation, rather than simple course - to - course articulation, will create 
educational pathways that ensure Tech-Prep students experience integrated 
academic and vocational technical learning, seamlessly connected through 
secondary and postsecondary institutions, so they are prepared for real careers in 
the California economy. 

The record of past accountability of the existing consortia - with respect to both 
student performance outcomes and organizational viability and effectiveness - lacks 
consistency. During the period July, 2000 through June, 2002, the funded consortia 
will be monitored with respect to their attainment of the accountability standards 
stated elsewhere in this Plan, and with respect to their ability to provide sufficient 
scope of career paths to compare favorably with their regional counterparts. Tech- 
Prep Local Consortia found to be deficient in either respect will be given incentives 
to consolidate with other Tech-Prep Local Consortia in their regions. 

Lunding for Tech-Prep Local Consortia will be determined based on the following 
criteria: 

1. Successful past performance. 

2. Comprehensive inclusion of high schools and community colleges within the 
consortium's geographic boundaries. 

3. Attainment of performance targets set by the state. 

4. Articulation agreements in place, functioning, and reviewed annually that 
include at least two years of secondary school preceding graduation, and two 
years or more of higher education, or an apprenticeship program of at least two 
years. 
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5. Submission of a Tech-Prep Local Consortium strategic plan meeting established 
state and regional priorities. 

6. Programs linked to identified current and future needs of business and industry, 
that are reviewed and validated annually. 

7. Evidence of integrated academic and vocational and technical curricula 
throughout secondary and postsecondary levels. 

8. Evidence of a system to track student performance and programmatic data 
through the secondary and postsecondary levels. 

9. Evidence of strategies to ensure equal access for all student populations. 

The Tech-Prep Local Consortia will be responsible for the following functions: 

• Improve existing Tech-Prep Programs to ensure curriculum integration, trained 
staff, functional articulation agreements, and student completion, including at 
risk students and those with special needs. 

• Develop new Tech-Prep Programs to address the new and emerging 
employment needs and opportunities locally and throughout the state. 

• Provide regular opportunities for secondary and postsecondary faculty to meet 
and confer regarding all aspects of the instructional program. 

• Provide and participate in on-going staff development to ensure understanding 
of curriculum development, instructional strategies, proven practices and 
innovative approaches, and technology, business and industry conditions, and 
future trends. 

• Provide incentives for innovative initiatives including new program concepts, 
new technologies, alternative assessment strategies, distance learning, and 
learning partnerships with business and industries. 

• Assure attainment of established state performance indicators and measures. 

• Submit annual reports on the performance of Tech-Prep Local Consortia. 

• Develop and update programmatic articulation agreements that foster student 
transition and achievement, from secondary through postsecondary, including 
baccalaureate degree programs, as appropriate. 

• Coordinate Tech-Prep with other Carl D. Perkins funded programs, and other 
education and workforce preparation efforts within the consortium and region. 

Industry-based Career Paths 

The recent, fundamental shift of the California economy to a foundation in 

technology and service has radically re-defined the skills sets required of the 

workforce. To the extent that the public schools and colleges are the primary 

providers of workforce preparation, it is imperative they direct their resources 
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toward ensuring their curricula do indeed prepare students for real jobs in the real 
industries that comprise the economy. 

Acting on that common imperative in separate ways, the California Department of 
Education has identified the most dynamic industries in the new economy that are 
high employing and those that have high employing potential. 

(a) Agriculture and Natural Resources 

(b) Arts, Media and Entertainment 

(c) Building Trades and Construction 

(d) Energy and Utilities 

(e) Engineering and Design 

(f) Fashion and Interiors 

(g) Finance and Business 

(h) Health and Human Services 

(i) Hospitality, Tourism and Recreation 

(j) Information Technology 

(k) Manufacturing and Product Development 

(l) Public and Private Education Services 

(m) Public Services 

(n) Retail and Wholesale Trade 

(o) Transportation 

The California Community Colleges have identified new and emerging technologies 
to which they are allocating resources. They are: 

1. Advanced Transportation Technologies 

2. Applied Competitive Technologies 

3. Biotechnologies 

4. Business and Workforce Performance Improvement 

5. Environmental Technologies 

6. Health Care Delivery 

7. Information Technologies 

8. International Trade Development 

9. New Media /Multimedia /Entertainment 

10. Small Business Development 

11. Engineering /Industrial Technologies (Proposed) 

During the period July, 2000 through June, 2002, the Chancellor's Office and the 
California Department of Education, with assistance from intersegmental faculty 
and staff already involved with the Career Majors initiative, will use these two lists 
as a mechanism for refining articulation policies and processes and, as appropriate, 
will identify other growth sections of the economy. 

• Tech-Prep Local Consortia, as described in the Act, will be invited to develop 
articulated curriculum models, multi-measure assessment, and staff 
development for the various industry-based career paths. 
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• Tech-Prep Local Consortia, as described in the Act, will be eligible to apply for 
supplemental funds for innovative practices and programs. 

• Tech-Prep Local Consortia, as described in the Act, may apply for supplemental 
funding to accomplish this work. 1 


1 Details for applying are in the Local Consortia Application. 
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